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Word reordering is one of the most difficult aspects of statistical machine translation (SMT), 
and an important factor of its quality and efficiency. Despite the vast amount of research 
published to date, the interest of the community in this problem has not decreased, and no single 
method appears to be strongly dominant across language pairs. Instead, the choice of the optimal 
approach for a new translation task still seems to be mostly driven by empirical trials. 

To orientate the reader in this vast and complex research area, we present a comprehensive 
survey of word reordering viewed as a statistical modeling challenge and as a natural language 
phenomenon. The survey describes in detail how word reordering is modeled within different 
string-based and tree-based SMT frameworks and as a stand-alone task, including systematic 
overviews of the literature in advanced reordering modeling. 

We then question why some approaches are more successful than others in different language 
pairs. We argue that, besides measuring the amount of reordering, it is important to understand 
which kinds of reordering occur in a given language pair. To this end, we conduct a qualitative 
analysis of word reordering phenomena in a diverse sample of language pairs, based on a 
large collection of linguistic knowledge. Empirical results in the SMT literature are shown to 
support the hypothesis that a few linguistic facts can be very useful to anticipate the reordering 
characteristics of a language pair and to select the SMT framework that best suits them. 


1. Introduction 


Statistical machine translation (SMT) is a data-driven approach to the translation of 
text from a natural language into another. Emerged in the 1990s and matured in the 


2000's to become widespread today, the core SMT methods (Brown et al. 1990 1993 
Berger et al. 1996 Koehn, Och, and Marcu 2003[l learn direct correspondences between 


source and target language from collections of translated sentences, without the need of 
abstract linguistic representations. The main advantages of SMT are versatility and cost- 
effectiveness: in principle, the same modeling framework can be applied to any pair 
of languages with minimal engineering effort, given sufficient amount of translation 
data. However, experience in a diverse range of language pairs has revealed that this 
form of modeling is highly sensitive to structural differences between source and target 
language, particularly at the level of word order. 
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SRC verb 

subject 

object 

complement 

jdd 

renewed 

AlEAhl Almgrby Almik mHmd AlsAds 

the-monarch the-Moroccan the-King Mohamed the-sixth 

dEm -h 

support his 

1- m$rwE Alrjys Alfrnsy 

to project the-president the-French 

REF The Moroccan monarch King Mohamed VI renewed his support to the project of the French President 


Figure 1: Arabic source sentence (right-to-left) and English reference franslafion, faken 
from fhe NIST-MT09 benchmark. The Arabic senfence is morphologically segmenfed by 
AMIRA (Diab, Hacioglu, and Jurafsky 2004) according fo fhe Arabic Treebank scheme, 
and provided wifh Buckwalter transliferafion (leff-fo-righf) and English glosses. 


Indeed, nafural languages vary greafly in how fhey arrange senfence componenfs, 
and franslafing words in fhe correcf order is essenfial fo preserve meaning across lan¬ 
guages. In English, for insfance, fhe role of differenf predicafe argumenfs is defermined 
precisely by fheir relafive posifion wifhin fhe senfence. Consider fhe franslafion exam¬ 
ple in Eigure Looking af fhe English glosses of fhe Arabic senfence, one can see fhaf 
corresponding words in fhe two languages are placed in overall similar orders with the 
notable exception of fhe verb (jdd/renewed), which occurs af fhe beginning of fhe Arabic 
senfence buf in fhe middle of fhe English one — more specifically, befween fhe subjecf 
and fhe objecf. To reach fhe correcf English order, fhree ofher reorderings are required 
befween pairs of adjacenf Arabic words: {AlEAhl/the-monarch, Almgrby/the-Morocean), 
{dEm/support, -h/his) and {Alrlys/the-president, Alfrnsy/the-French). This example suggesfs 
a simple division of reordering pafferns info long-range, or global, and shorf-range, or 
local. However ofher language pairs display more complex, hierarchical patterns. 

Word reordering phenomena are nafurally handled by human franslafor^buf are 
a major source of complexify for SMT. In very general ferms, fhe fask of SMT consisfs 
of: breaking fhe inpuf senfence into smaller units, selecting an optimal translation for 
each unif and placing fhem in fhe correcf order. Searching for fhe overall besf frans¬ 
lafion fhroughouf fhe space of all possible reorderings is, however, compufafionally 
intracfable i Knighf 1999) . This crucial facf has mofivafed an impressive amounf of 
research around two inter-related questions: namely, how to effectively restrict the set 
of allowed word permutations? and how to detect the best permutation among them? 

Existing solutions to these problems range from heuristic constraints, based on 
word-to-word distances and completely agnostic about the sentence content, to linguis¬ 
tically motivated SMT frameworks where the entire translation process is guided by 
syntactic structure. The research in word reordering has advanced together with the 
core SMT research and has sometimes directed it, being one of the main motivation 
for the development of tree-based SMT. At the same time, the variety of word orders 
existing in the world languages has pressed the SMT community to admit the impor¬ 
tance of language-specific knowledge and to reassess its ambitions towards a universal 
translation algorithm. 

According to the Machine Translation Archive, a scientific interest in this specific 
subproblem of MT started around 2006 and kept growing at a rapid pace. In 2014, the 
research papers mainly dedicated to reordering accounted for no less than 10% of all 


1 Nevertheless learning and understanding a new language has b een shown to be more difficult when the 
new language is structurally distant from one's native language jCorder 1979|. 
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SMT papersj^Despite the abundant research, word order differences remain among fhe 
mosf imporfanf facfors of performance in modern SMT sysfems, and new approaches 
fo reordering are sfill proposed every year. 

To orienfafe fhe reader in fhis complex and productive research area, we presenf a 
comprehensive survey of word reordering viewed as a sfafisfical modeling challenge 
and as a nafural language phenomenon. Our survey nofably differs from previous 
(|Cosfa-jussa and Fonollosa 2009 f in fhaf we do nof only review fhe existing approaches 
fo word reordering in SMT, buf we also question why some approaches are more 
successful fhan ofhers in differenf language pairs. In particular, we argue fhaf under- 
sfanding fhe complexify of reordering in a given language pair is key fo selecfing fhe 
righf SMT models and fo improving fhem. 

The survey is organized as follows: Section explains how fhe word reordering 
problem is freafed wifhin differenf sfring-based and free-based SMT frameworks, as 
well as a sfand-alone fask (i. e. pre- and posf-ordering). The liferafure in advanced 
reordering modeling is exfensively reviewed, wifh a major focus on recenf work. Section 
[^describes fhe challenges of aufomafically assessing word reordering accuracy in SMT 
outpufs. Secfion presenfs a qualifafive analysis of word reordering across language 
pairs. In particular, defailed word order profiles are provided for a sample of seven 
widely spoken languages representing sfrucfural and geographical diversify: namely 
English, German, French, Arabic, Turkish, Japanese and Chinese. The same secfion 
reviews empirical resulfs from fhe SMT liferafure showing fhaf fhe proposed word 
order profiles are useful fo anficipafe fhe reordering characferisfics of a language pair 
and fo selecf fhe SMT framework fhaf besf suifs fhem. The survey ends wifh a discussion 
of fhe sfrengfhs and weaknesses of fhe major approaches fo reordering in SMT. 

2. Approaches to Word Reordering in Statistical Machine Translation 

A first important distinction has to be made between word reordering performed as parf 
of fhe decoding process (Sections |2.1| fo |2.3| and word reordering performed before or 
after if as a monolingual fask decoupled from fhe bilingual franslafion fask (Secfion |2.4[ |. 

Wifhin fhe former, we furfher distinguish befween sfring-based (sequential) ap¬ 
proaches and free-based (sfrucfural) approaches. String-based SMT (Sections |2.1| and 
|2.2[ treats translation as a sequential task: the target sentence is built from leff fo righf 
while fhe inpuf unifs are visifed in differenf orders and no dependencies ofher fhan 
word adjacency are considered. Subsequenfly problem decomposition is applied fo fhe 
fargef string: an opfimal franslafion is soughf for each prefix of fhe fargef franslafion, 
from fhe shorfesf fo fhe longesf. Tree-based SMT (Secfion |2.3[ l posifs fhe exisfence of a 
tree structure fo explain franslafion as a hierarchical process and fo capfure dependen¬ 
cies among non-adjacenf fexf unifs. Problem decomposition is fherefore based on fhis 
sfrucfure: an opfimal franslafion is soughf for each word span corresponding fo a node 
in fhe free, from fhe leaves up fo fhe roof. Whereas sfring-based SMT has fo search over 
all inpuf permufafions fhaf do nof violafe some general reordering consfrainfs, free- 
based SMT considers only fhose permufafions fhaf resulf from fransforming a given 
free representing fhe inpuf senfence (as for example permuting each node's children). 

Moreover, we should nofe fhe difference befween syntax-based SMT approaches 
that utilize trees produced by monolingual parsers trained on S 5 mtactic treebanks, and 


2 Peer-reviewed conferences, workshops and journal papers listed by the Machine Translation Archive: 
http://www.mt-archive.info/srch/subjects.htm 
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data-driven tree-based SMT approaches that extract bilingual translation grammars 
directly from pairs of source and fargef senfences. In fhe former, word reordering is 
consfrained by a given synfacfic parse free of fhe inpuf senfence or by fhe grammar of 
fhe fargef language (or bofh), whereas, in fhe laffer, free sfrucfure capfures hierarchical 
reordering pafferns fhaf may or may nof correspond fo S5mfacfically mofivafed rules. 

In general, fhe SMT search (or decoding) process consisfs in searching for fhe mosf 
probable fargef (or English) senfence e* given a source (or foreign) senfence f by scoring 
franslafion h 5 rpofheses fhrough a linear combination of feafure functions: 


e = arg max max exp 

e b 


R 


Arhr(f, e, b) 


( 1 ) 


where b is a lafenf variable representing eifher a linear or a hierarchical mapping 
(alignmenf) befween f and e, hr(e,f,b) are R arbifrary feafure funcfions and A,- the 
corresponding feature weights. Feature functions try to capture relevant translation 
adequacy and word reordering aspects from aligned parallel dafa, as well as franslafion 
fluency aspecfs from monolingual fargef fexfs. Moreover, feafure funcfions are assumed 
fo be locally decomposable fo allow for efficienf decoding via dynamic programming. 
Feafure weighfs are funed discriminafively by direcfly optimizing franslafi on qualif)!^ 
on a developmenf sef, using paramefer funing fechniques such as MERT ( Och 2003) , 
MIRA ( jChiang, Marfon, and Resnik 2008[ l or PRO ( [Hopkins and May 2011| . 


2.1 Phrase-based SMT 


Phrase-based SMT (PSMT) is fhe currenfly dominanf approach in sfring-based SMT. 
PSMT ruled ouf fhe early word-based SMT framework (Brown ef al. 1990 1993 Berger 
ef al. 19^ fhanks fo two imporfanf novelties: fhe use of mulfi-word franslafion unifs 
( jOch 199^ Zens, Och, and Ney 2002{ Koehn, Och, and Marcu 20031, and fhe move from 
a generative fo a discriminative modeling framework (Och and Ney 2002|. 

The search process ([^ in PSMT is guided by fhe fargef sfring e, builf from leff fo 
righf, and fhe alignmenf variable b which embeds bofh segmenfafion and reordering of 
fhe source phrases. This is defined as: 


h^bi = ((Ji, Al), (J2, A 2 ),..., {Jj, Ki)) 


( 2 ) 


such fhaf Al,..., Kj are consecutive infervals parfifioning fhe fargef word posifions, 
and Ji,... ,Ji are corresponding buf nof necessarily consecutive infervals parfifioning 
fhe source word posifions. A phrase segmenfafion for our running example is shown in 
Figure]^ 

The use of phrases mainly resulfs in a better handling of ambiguous words and 
many-fo-many word equivalences, buf if also makes if possible fo capfure a consid¬ 
erable amounf of local reordering phenomena wifhin a franslafion unif (infra-phrase 
reordering). Wifh reference fo our running example (Figure [^, a PSMT model may 
handle fhe local reorderings as single phrase pairs — [AlEahl AlmgrbyMThe Moroccan 
monarch] efc. — if fhese were observed in fhe framing dafa. On fhe confrary if is unlikely 
fhaf a single long phrase spanning from jdd fo AlsAds was observed, fherefore fhe long- 
range reordering of fhe verb could be handled by fnfer-phrase reordering. 
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Automatic measures of translation quality are discussed in 


Section|^ 
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OddJa [AlEAhl Almgrby], [Almik mHmd AlsAdsJj [dEm-h]4 [I-mSrwElj [Alr}ys Alfrnsyle 
[thelviorocc^^^^rchli^^Khi^^Wlo^^ied^^Tffi^wedla [hi^^s^port]4 [to the'^ject ofjj [the French Presidentlg 


Figure 2: An example of word alignment and phrase segmentation for the sentence pair 
presented in Figure]^ Subscript indices denote the phrase alignment b{. Note that other 
phrase segmentations are possible given the same word alignment. 


State-of-the art PSMT systems t 5 rpically include the following core feature functions: 
phrase- and word-level translation models; target n-gram language model; distortion 
penalty; plus additional components that model specific translation aspects. Assuming 
a one-to-one correspondence between source and target phrases, reordering in PSMT 
means searching through a set of permutations of the source phrases. Thus, two sub¬ 
problems arise: defining the set of permutations in b allowed during decoding (reorder¬ 
ing constraints) and scoring the allowed permutations (reordering models or feature 
functions). We will now discuss each of them in detail. 


2.1.1 PSMT reordering constraints. Because searching over the space of all possible 
translations is NP-hard ( [Knight 1999) , SMT decoders employ heuristic search algo¬ 
rithms to only explore a promising subset of the search space. In particular, limiting the 
set of explorable input permutations is an essential way to reduce decoding complexity. 

The reordering constraint originally included in the PSMT framework is called 
distortion limit (DL). This consists in allowing the decoder to jump, or skip, at most 
k words between the last translated source phrase and the next one, i.e.: 

jump{Ji-i, Ji) = \start{Ji) — end{Ji-i) — 1\ < k (3) 


Setting a low distortion limit means only exploring local reorderings, based on the 
arguable assumption that languages tend to arrange sentence constituents in similar 
orders. Besides being essential for efficiency — DL allows for linear decoding com¬ 
plexity —, reordering constraints are also important for translation quality because the 
existing SMT models are t 5 rpically not discriminative enough to guide the search over 
very large sets of reordering h 5 rpotheses. However, reordering constraints have also 
several drawbacks. For instance the verb reordering in Figure|^may not be captured by 
a PSMT system that applies a DL of k=5 or less, because jumping back from AlsAds to 
jdd corresponds to a skip of 6 positions. While the distortion limit is a de facto standard 
in modem PSMT systems, the first constraining paradigms were formulated earlier for 
word-based SMT (Berger et al. 1996||Zens and Ney 2003| and are called IBM constraints. 

A different kind of reordering constraint can be derived from the Inversion Trans¬ 
duction Grammars (ITG) ( jWu 1995 1997| |. ITG constraints only admit permutations 
that are generated by recursively swapping pairs of adjacent blocks of wordsj^ In 
particular, ITG constraints disallow reorderings that generalize the patterns (314 2) 
and (2 41 3), which are rarely attested in natural languages (Wu 1997 1 ^Enforcing ITG 
constraints in left-to-right PSMT decoding requires the use of a shift-reduce permuta¬ 
tion parser (jZens 2008 Feng et al. 2010|. Alternatively, a relaxed version of the ITG 


4 For a comparative study of the IBM and ITG constraints, we refer the reader to 

Zens and Nev (2003 

5 Empirical evidence against this was presented by 

Wellington, Waxmonsky, and 

Melamed {2UU6 . 
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constraints (i.e. Baxter permutations) may be enforced by simply inspecting the set 
of covered source positions, as proposed by Zens ef al. (2004) and Zens (2008) . Infer- 
esfingly Cherry, Moore, and Quirk (2012| found no consisfenf benefif from applying 
eifher exacf or approximafe ITG-consfrainfs fo a PSMT sysfem fhaf already included a 
hierarchical phrase orienfafion mode|^(|Galley and Manning 2008}. 

The reordering consfrainfs presenfed so far are nof sensifive fo fhe words being 
franslafed nor fo fheir confexf. This resulfs in a very coarse definition of fhe reordering 
search space, which is problematic in language pairs wifh differenf S 5 mfacfic sfrucfures. 
To address fhis problem, Yahyaei and Monz (200^ propose fo decouple local and 
global reordering by segmenting fhe inpuf senfence info chunks fhaf can be permufed 


arbifrarily, buf each of which is franslafed monofonically. In a relafed work, Yahyaei and 
Monz (2010[| presenf a fechnique fo dynamically set the DL during decoding: they train 


a discriminative classifier fo predicf fhe mosf probable jump lengfh affer each inpuf 
word, and use fhe predicfed value as fhe DL affer fhaf posifion. Unforfunafely, fhis 
mefhod appears fo generafe inconsisfenf consfrainfs leading fo decoding dead-ends. 


Bisazza and Federico (2013a I furfher develop fhis idea so fhaf only long reorderings 


predicted by a specific reordering model are explored by fhe decoder. This form of early 
reordering pruning enables fhe PSMT sysfem fo capfure long-range reordering wifhouf 
hurling efficiency and is nof affecfed by fhe consfrainf inconsisfency problem. 

When available, a parse free of fhe inpuf may also be used fo consfrain PSMT 
reordering following fhe principle of syntactic cohesion (|Fox 2002||. Concretely, the 
dependency cohesion constraint (jCherry 2008| states that, when part of a source subfree 
is franslafed, all words under fhe same subfree musf be covered before moving fo 
words oufside of if. Infegrafed in phrase-based decoding as soff consfrainfs — i. e. 
by using fhe number of violafions as a feafure funcfion — dependency cohesion and 
ifs varianfs (Cherry 2008 Bach, Vogel, and Cherry 2009 ) were shown f o significanfly 
improve franslafion qualify. In a relafed work,|Feng, Sun, and Ney (2012|| derive similar 
cohesion consfrainfs from fhe semanfic role labeling sfrucfure of fhe inpuf senfence. 
The divide-and-franslafe approach of Sudoh ef al. (2010) employs source-side parse 
frees fo segmenf complex senfences info simple clauses which are replaced by specific 
symbols and franslafed independenfly. Then, fhe fargef senfence is reconsfrucfed using 
fhe placeholders, wifh fhe aim of simplifying long-range clause-level reordering. 


2.1.2 PSMT reordering feature functions. Target language modeling is the primary way 
to reward promising reorderings during translation. This happens indirectly, through 
the scoring of fargef word n-grams fhaf are generafed by franslafing fhe source positions 
in differenf orders. Flowever, fhe fixed-size confexf of language models used in SMT 
(fypically 4 or 5 words) makes fhem largely insensitive fo global reordering phenomena. 
In fhe lasf years, a growing inferesf for language pairs wifh very differenf word orders, 
such as Arabic-English and Chinese-English, has favored fhe developmenf of new fech- 
niques fo explicifly model fhe reordering problem. Given a source senfence, fhe search 
for ifs optimal reordering is generally decomposed info a sequence of local reordering 
decisions, as is done for fhe whole franslafion process. Thus, fhe basic reordering sfep 
corresponds fo fhe relative positioning of fhe word, or phrase, being franslafed wifh 
respecf fo fhe word, or phrase, fhaf was previously franslafed. 

The simplesf example of reordering feafure funcfion is fhe distortion cost or distor¬ 
tion penalty jump{Ji-i, Ji), which by convention assigns zero cost to h 5 q)otheses that 


6 The reordering models mentioned herein are explained in detail in the next subsection. 
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Reordering models 

References 

Model 

type 

Reordering step 
classification 

Features 

Phrase orientation 

Example: P{oTient=discontinuous-left 

nodels (POM): 

next-phrase-paiT=[jdd]-[renewed]) 

lexicalized (hierarchical) 
phrase orientation model 

Tillmann 2004; 

Koehn & al. 2005; 

Nagata & al. 2006; 

Galley & Manning 2008 

gener. 

monotonic, swap, 
discontinuous 
(left or right) 

source/target phrases 

phrase orientation 
maxent classifier 

Zens & Ney 2006 

discr. 

source/target words 
or word clusters 

sparse phrase 
orientation features 

Cherry 2013 

discr. 

Jump models (JM): 

Example: P(jump=—5 {rom=AlsAds, to=jdd ) 

inbound/outbound/pairwise 
lexicalized distortion 

Al-Onaizan & Papineni 

2006 

gener. 

jump length 

source words 

inbound/outbound 
length-bin classifier 

Green c& al. 2010 

discr. 

jump length based 
(9 length bins) 

source words, POS, 
position; sent, length 

Source decoding sequence models (SDSM): 

Example: P(next-woid=jdd prev-tTanslated-words=AlEahil Almlk mHmd AlsAds) 

reordered source n-gram 

Feng & al. 2010a 

gener. 

— 

source words 
(9-gram context) 

source word-after-word 

Bisazza & Federico 2013; 

Goto & al. 2013 

discr. 

— 

source words, POS; 

source context's words 

and POS 

Operation sequence models (OSM): 

Example: P( next-operation=generate[jdd,renewed] prev-operations=generate[AlsAds,VI] jumpBack[l] ) 

translation/reordering 
operation n-gram 

Durrani & al. 2011; 
Durrani & al. 2013; 

Durrani & al. 2014 

gener. 

insertGap, 

jumpBack, 

jumpForward 

source/target words, 
POS or word clusters; 
prev. n -1 operations 


Table 1: An overview of state-of-the-art reordering models for PSMT. Model type 
indicates whether a model is trained in a generative or discriminative way. All examples 
refer to the sentence pair shown in Figure 


preserve the order of the source phrases (monotonic translations). During decoding, 
the basic implementation of distortion cost penalizes long jumps only when they are 
performed, leading to the proliferation of hypotheses with gaps (i. e. uncovered input 
positions). This issue can be addressed by incorporating into the distortion cost an 
estimate of the cost yet to be incurred ([Moore and Quirk 2007|. 


State-of-the-art systems use the distortion cost in combination with more sophisti¬ 
cated reordering models that take into account the identity of the reordered phrases and, 
optionally, various kinds of contextual information. A representative selection of such 
models is summarized in Table To ease the presentation, we have divided the models 
into four groups according to their problem formulation: phrase orientation models, 
jump models, source decoding sequence models and operation sequence models. 

Phrase orientation models (POM) (jTillmarm 2004} Koehn et al. 2005 Nagata et 


al. 2006 Zens and Ney 2006 Li et al. 26l4|, simply kiiown as lexicalized reordering 
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models, predict whether the next translated source span should be immediately to 
the right (monotone), immediately to the left (swap) or anywhere else (discontinuous) 
relatively to the last translated one|^ For example, in Figure the phrase pair [Alnilk 
mHmd AlsAdsMKing Mohamed VI] has monotone orientation whereas IjddMrenewed] 
has discontinuous left orientation with respect to the previously translated phrase. 
Because of fheir simple reordering sfep classification, POM can be conditioned on very 
fine-grained information, such as fhe whole phrase pair, wifhouf suffering foo much 
from dafa sparseness. However, since POM ignore fhe disfance befween consecutively 
franslafed phrases, fhey cannof properly handle long-range reordering phenomena and 
are f 5 rpically used with a low distortion limit. 


Jump models (JM) (Al-Onaizan and Papineni 2006 Green, Galley, and Manning 
2010[ | predict the direction and length of fhe jump fhaf is performed befween con¬ 
secutively franslafed words or phrases, wifh fhe goal of beffer handling long-range 
reordering. Due fo dafa sparseness, JM work besf when framed in a discriminative 
fashion using a variefy of binary feafures (such as fhe lasf franslafed word, ifs POS fag 
and relative position in the sentence) and when length bins are used instead of fhe exacf 
jump lengfh (|Green, Galley, and Manning 2010 1 . A drawback of JM is fhaf fhey fypically 
over-penalize long jumps because fhey are more rarely observed fhan shorf jumps. 

Source decoding sequence models (SDSM) address fhe said issue by direcfly mod¬ 
eling fhe reordered sequence of inpuf words, as opposed fo fhe reordering operations 
that generated it. This in turn can be done in several ways, such as: training n-gram 
models on target-like reordered source sentences and using them to score the sequence 
of inpuf words visifed by fhe decoder ( [Feng, Mauser, and Ney 2010 1; fagging fhe whole 
inpuf senfence wifh symbols denoting how each word should be reordered wifh respecf 
fo ifs leff and righf confexf, fhen rewarding fhe decoding pafhs fhaf mosf agree wifh fhe 
fag sequence (Feng, Pefer, and Ney 20131; and finally, predicting which inpuf posifion is 
likely fo be franslafed right after a given input position by means of a maximum enfropy 
model using word and confexf feafures ( [Bisazza and Federico 2013a Goto ef al. 2013| |. 

Operation sequence models (OSM) ^urrani, Schmid, and Fraser 201 Ij l are n-gram 
models that include lexical translation operations and reordering operations (insertGap, 
jumpBack or jumpForward ) in a single generative story, thereby combining elements 
from fhe previous fhree model families. An operation sequence example is provided in 
fhe lower parf of Table OSM are closely relafed fo n-gram based SMT models (see 
nexf section) buf have been successfully applied as feafure functions fo PSMT (Durrani 
ef al. 201^ . To overcome dafa sparseness, OSM can be successfully applied fo POS-fags 
and unsupervised word clusters ( [Durrani ef al. 2014) . 

SDSM and OSM have been proven optimal for language pairs where high disforfion 
limifs are required fo capfure long-range reordering phenomena ( [Durrani, Schmid, and 
Fraser 2011|[Bisazza and Federico 2013b| | Gofo ef al. 2013^ . Neverfheless POM remains 
the most widely used t 5 q)e of phrase-based reordering model and is considered a nec¬ 
essary componenf of PSMT baselines in any language pair. In particular, fwo varianfs 
of POM deserve furfher attention because of fheir nofable effecf on franslafion qualify: 
hierarchical POM and sparse phrase orienfafion feafures. 

Hierarchical phrase orienfafion models, or simply hierarchical reordering models 
(HRM) ( [Galley and Marming 20(i8| improve fhe way in which fhe orienfafion of a new 
phrase pair is determined: already translated adjacent blocks are merged together to 


7 Some phrase orientation models further distinguish between discontinuous left and discontinuous right. 
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standard: discontinuous 


F 

hierarchical: swap 



Oddis 

[AlEAhl Almgrby], 

[Almik mHmd AlsAdsJs 




[dEm-h]4 ... 


[the Moroccan monarch]^ [King Mohamed Vljg [renewedjj [his support]4 


Figure 3: Phrase orientation example for the phrase pair [jdd]-[renezued]: the standard 
model detects a discontinuous orientation with respect to the last translated phrase (2) 
whereas the hierarchical model detects a swap with respect to the block of phrases (1-2). 


form longer phrases around fhe currenf one. For insfance in Figure HRM merges 
phrases 1 and 2 info a large phrase pair [AlEahl... AlsAdsMThe ... VI] and consequenfly 
assigns a swap, insfead of discontinuous orienfafion, fo [jdd]-[renezved]. As a resulf, ori- 
enfafion assignmenfs become more consisfent across h 5 rpofheses wifh differenf phrase 
segmenfafions. 

Rafher fhan training a reordering model by relative frequency or maximum enfropy 
and using ifs score as one dense feafure function. Cherry (2013| infroduces sparse 
phrase orientation features that are directly added to the model score during decoding 
(cf. equation Q) and optimized jointly with all other SMT feature weights. Effective 
sparse reordering features can be obtained by simply coupling a phrase pair's orienta¬ 
tion with the first or last word (or word class) of ifs source and targef side ( Cherry 2013 [ I, 
or even wifh fhe whole phrase pair identify (|Auli, Galley, and Gao 201^. 


2.2 N-gram based SMT 


N-gram based SMT (Casacuberfa and Vidal 2004 Marino ef al. 2006|l is a sfring-based 
alfernafive fo PSMT. In fhis framework, smoofhed n-gram models are learnf over se¬ 
quences of minimal translation units (called tuples), which, like phrase pairs, are pairs 
of word sequences exfracfed from word-aligned parallel senfences. Tuples, however, are 
fypically shorfer fhan phrase pairs and are exfracfed from a unique, monotonic segmenfa- 
fion of fhe senfence pair. Thus, fhe problem of spurious phrase segmenfafion is avoided 
buf non-local reordering becomes an issue. For insfance, in Figure 1^ a monofonic 
phrase segmenfafion could be achieved only by freafing fhe large blockfydd ... AlsAds]- 
[The ... renewed] as a single fuple. Reordering is fhen addressed by Tuple unfolding' 
( |Crego, Marino, and de Gisperf 2005| : fhaf is, during framing fhe source words of each 
franslafion unif are rearranged in a fargef-like order so fhaf more, shorfer fuples can be 
exfracfed. Af fesf fime, inpuf sentences have to be pre-ordered for franslafion. To fhis end, 
Crego and Marino (2006| propose fo precompufe a number of likely permufafions of fhe 
inpuf using POS-based rewrife rules learned during fuple unfolding. The reorderings 
fhus obfained are used fo exfend fhe search graph of a monofonic decoderj^Reordering 
is offen considered as a shorfcoming of n-gram based SMT as reordering decisions are 
largely decoupled from decoding and mosfly based on source-side informafion. 


8 More pre-ordering techniques will be discussed in Section 
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2.3 Tree-based SMT 


The SMT frameworks discussed so far learn direcf correspondences between source 
and fargef words or phrases, freafing reordering as a sequenfial process. This flaf repre- 
senfafion is fairly successful for some language pairs however, in ofhers, reordering is 
more nafurally described as a hierarchical process where small, locally reordered blocks 
become fhe elemenfs of recursively larger reordered blocks. Concrefely, in our running 
example (Figure|^, a hierarchical or free-based approach would make if possible fo: firsf 
franslafe and reorder small blocks such as [AlEahl Almgrby] and [Almlk mHmd AlsAds], 
fhen merge fhem fo compose a larger block fhaf gefs reordered as a whole wifh respecf fo 
fhe verb jdd, and so forfh. The degree of generalization af each level would fhen depend 
on how blocks are represenfed: e.g. by fheir lexical confenf, by a fag denofing fhe block's 
synfacfic cafegory, or by a generic symbol. 

Tree-based approaches are largely inspired by S 5 mfacfic parsing, buf nof all in fhe 
same way: some model franslafion as fhe fransformafion of frees produced by monolin¬ 
gual parsers framed on S 5 mfacfic freebanks (Section 2.3. 1|, while ofhers exfracf a bilin¬ 
gual franslafion grammar direcfly from word-aligned parallel fexf wifhouf using any 
synfacfic information (Secfion |2.3.2[ |. Non-S 5 mfacfic bilingual franslafion grammars may 
sfill be enriched wifh S 5 mfacfic information, for insfance in fhe form of soff consfrainfs 
(Secfion |2.3.3t . 

All free-based frameworks crucially differ from PSMT and ofher sfring-based 
frameworks wifh respecf fo reordering: Whereas PSMT considers all inpuf permufafions 
fhaf do nof violafe general reordering consfrainfs and fhen scores fhem wifh separafe 
reordering models, free-based sysfems model reordering joinfly wifh franslafion and, 
during decoding, only (or mosfly) explore inpuf permufafions fhaf are licensed by fhe 
learnf franslafion model. 

Mosf modem free-based approaches fall under fhe general formulation of SMT 
which scores franslafion h 5 q)ofheses by a linear combination of feafure funcfions (see 
equation Q), wifh a franslafion model (or grammar) and a fargef language model as 
core feafures. Tree-based decoding is usually performed by a charf parsing algorifhm 
wifh beam search and infegrafed fargef language model. Hence, fhe fargef senfence is 
nof produced from leff fo righf as in sfring-based SMT, buf boffom-up according fo a 
free derivation order. 


2.3.1 Syntax-based SMT. An imporfanf motivation for using S 5 mfax in SMT is fhaf 
reordering among nafural languages very offen involve fhe permufafion of whole 5501 - 
facfic consfifuenfs (see e.g. |Fox (2002[ |). For insfance, in our running example (Figure]^, 
knowing fhe span of fhe Arabic subjecf would be enough fo predicf fhe reordering of 
fhe verb for franslafion info English. 

S 5 mfax-based SMT encompasses a variety of frameworks fhaf use synfacfic annofa- 
fion eifher on fhe source or on fhe fargef language, or fo bofh. So-called tree-to-string 
mefhods ( [Huang, Knighf, and Joshi 2006 Liu, Liu, and Lin 2006| use a given inpuf sen- 
fence parse free fo resricf the application of franslafion/reordering rules fo word spans 
fhaf coincide wifh S 5 mfacfic consfifuenfs of specific cafegories. For insfance, fhe swap of 
Alrjys Alfrnsy may only be dicfafed by a rule applying fo noun phrases composed of a 
noun and an adjective. On fhe ofher han d, string-to-tree mefhods ([Ya mada and Knighf 


2002 Galley ef al. 2004 Marcu ef al. 2006 Shen, Xu, and Weischedel 20lO| employ S 3 mfax 

as a way fo resfricf franslafion h 5 q)otheses fo well-formed fargef language senfences — 
ruling ouf, for insfance, a franslafion fhaf fails fo reorder fhe franslafed verb renewed 
wifh respecf fo ifs subjecf. Using S 5 mfax on bofh source and fargef sides (tree-to-tree) 
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VBD NN JJ NN NNP JJ-NUM NN PRP IN NN NN JJ 

jdd AlEAtil Almgrby Almik mHmd AlsAds dEm -h I- m$rwE Alr}ys Alfrnsy 



Moroccan monarch King Mohamed VI renewed his support to the project of the French President 


Treelet pairs: 


AlEAhl 


I 



AlEAhl Almgrby 


jdd AlEAhl 



jdd dEm 


monarch Moroccan monarch monarch renewed Moroccan monarch renewed renewed support 

(1) (2) (3) (4) (5) 

Order templates: 

VBD, NN 2 NN 3 ^ 2 1 3 

(a) 


NN, JJ 2 NN 3 ^ 2 1 3 

(b) 


NN, NNPj JJ-NUM 3 ^ 1 2 3 

(C) 


Figure 4: Examples of treelet pairs and order templates extracted from a word-aligned 
sentence pair and its source-side dependency parse tree. The projected tree for the whole 
target sentence is not shown due to space limitations. 


jimamura, Okuma, and Sumita 2005 

Ding and Palmer 2005 

Smith and Eisner 2006 

Watanabe, Tsukada, and Isozaki 2006 

Zhang et al. 2008} has proven rather difficult in 

practice due to the complexity of aligning potentially very different tree topologies and 


to the large size of the resulting translation grammars. Moreover, the need for high- 
quality parsers in both language sides seriously limits the applicability of this approach. 

Syntax-based SMT approaches also differ in the formalism they use to represent the 
trees. Those based on phrase structure (constituency) grammars t 5 rpically comply to the 
principle that each translation/reordering rule should match a complete constituent, 
while those based on dependency grammars opt for a more flexible use of structure. 
For example, in string-to-dependency SMT ( Shen, Xu, and Weischedel 2010[ | rules can 
correspond to partial constituents but must be either a single rooted tree, with each 
child being a complete sub-tree, or a sequence of siblings, each being a complete sub¬ 
tree. Partial dependency rules are then combined during decoding, which means that 
not all reordering decisions are governed by the translation model. 

An even more flexible use of structure is advocated by the treelet-based SMT 
framework (Quirk, Menezes, and Cherry 2005|, where translation rules can correspond 
to any connected subgraph of the dependency tree (i.e. treelet). As illustrated by Fig¬ 
ure]^ treelet pairs are extracted from pairs of source dependency parse tree and target- 
side projected tree. Treelets can be seen as phrases that are not limited to sets of adjacent 
words, but rather to sets of words that are connected by dependency relations, which 
in turn makes it possible to learn non-local reordering patterns. As reordering decisions 
are only partially governed by the translation model, treelet-based SMT benefits from 
additional model components specifically dedicated to reordering. For example, in 
Figure |4 treelet pair (3) determines the swapping of jdd and AlEAhl but does not 
specify the ordering of dEm which is also a child of jdd. Hence, during decoding, all 
possible reorderings of the urunatched children are considered and scored by a separate 
discriminative model predicting the position of a child node (or modifier m) relative to 
its head h, given lexical, POS and positional features of m and h. Reordering modeling 
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Hierarchical rules: 

(1) X ^ jdd XII X renewed 

(2) X ^ AlEAhl X, Almik X^ II the X, monarch King X^ 

(3) X ^ X -h II his X S 

(4) X ^ I- m$rwE X II to the project of the X s' ' 




the Moroccan monarch King Mohamed VI renewed his support to the project of the French President 


Regular phrase translation rules: 

(7) X Almgrby II Moroccan 

(8) X mHmd AlsAds II Mohamed VI 

(9) X A dEm II support 

(10) X Alr}ys Alfrnsy II French President 


Figure 5: Possible derivation of a word-aligned sentence pair and corresponding hier¬ 
archical phrase-based translation grammar. The target-side tree is not represented due 
to space limitations. 


is thus largely decoupled from lexical selection, which makes the model very flexible 
but results in a very large search space and high risk of search errors. To address this 
issue, Menezes and Quirk (2007| introduce another mechanism to complement treelet 


reordering: namely, dependency order templates. An order template is an unlexicalized 
rule specifying the reordering of a node and all its children based on their POS tags. 
For instance, in Figure treelet pair (3) may be combined with template (a) to specify 
the order of the child dEm. For each new test sentence, matching treelet pairs and order 
templates are combined to construct lexicalized translation rules for that sentence and, 
finally, decoding is performed with a chart parsing algorithm. 

We will now discuss SMT frameworks that model translation as a process of parallel 
parsing of the source and target language via a S 5 mchronous grammar. 

2.3.2 Tree-based SMT without syntax. The idea of extracting bilingual translation (i.e. 
synchronous) grammars directly from word-aligned parallel data originates in early 
work on Inversion Transduction Grammars (ITG) by Wu (1996 11997 1 . 


In a more mature approach, hierarchical phrase-based SMT (HSMT) (Chiang 


20051, the translation model is a probabilistic S 5 mchronous context-free grammar (SCFG) 
whose rules can correspond to arbitrary (i.e. non S5mtactically motivated) phrases la¬ 
beled by only two generic non-terminal symbols (X or S). As shown in Figure]^ FISMT 
translation rules can either include: a mix of terminals and non-terminals capturing 
reordering patterns and discontinuities (rules 1-4) or only terminals (7-10) basically 
corresponding to phrase pairs in string-based PSMT. Finally, the so-called glue rules (5- 
6) are always added to the grammar to combine translated blocks in a monotone fashion 
regardless of their content. As in PSMT, extracted translation rules may not exceed 
a certain length and rule scores are obtained using maximum likelihood estimation. 
Crucially, swapping adjacent phrases with no lexical evidence (x —X 1 X 2 IIX 2 X 1 ) is not 
allowed by standard HSMT grammars, therefore reordering can only be triggered by 
at least partially lexicalized translation rules. This is a major difference with respect to 
most S5mtax-based approaches where reordering can be captured by rules containing 
only labeled non-terminals (e.g. S —> NP VP || VP NP). This means that, for instance, 
the reordering pattern learnt by our example HSMT grammar (Figure rule 1) may 


12 


















Bisazza and Federico 


A Survey of Word Reordering in Statistical Machine Translation 


Regular phrase translation rules: 

(1) XO -> AlEAhl Almgrby II the Moroccan monarch 

(2) XO A Almik II King 

(3) XO A mHmd AlsAds II Mohamed VI 

(4) XO A dEm II support 
Monotonic block rules: 

(5) M2 ^ X0i XO 2 II XO, XO 2 

(6) M2 ^ XO M2 II XO M2 
Hierarchical rules: 

(7) Xt -> jdd M2 II M2 renewed 

(8) Xt AXO-hll hisXO 
Glue rules: 

(9) S A S Xt II S Xt 

(10) S A XI II XI 



AlsAds dEm -h 



the Moroccan monarch King Mohamed VI renewed his support 


Figure 6: Example of shallow-1 HSMT grammar with monotonic non-terminals M^. 
The target-side tree is not represented due to space limitations. 


only be used to reorder the specific verb form jdd (renewed) in subsequent test sentences. 
Thus, HSMT is likely to work better for languages where the syntactic role of phrases is 
mostly expressed by separate function words (e.g. Chinese) than for languages where 
this information is largely conveyed by word inflection (e.g. Russian). 

While hierarchical models are inherently capable of dealing with complex and 
recursive reordering patterns, in practice many translation rules are noisy or based on 
limited context. To limit search complexity, a constraint is imposed on the maximum 
number of source words that may be covered by a non-terminal symbol during decod¬ 
ing (span constraint). This parameter is t 5 rpically set to 10 or 15 words as wider spans 
result in prohibitively slow decoding and lower translation quality. For these reasons, 
a number of extensions to the original HSMT framework have been proposed with the 
specific goal of better handling complex reordering phenomena. 

Shallow-M grammars ( |de Gispert et al. 2010[ | can be used to refine the reordering 
space of HSMT according to the reordering characteristics of a specific language pair. 
For instance, as shown in Figure]^ an Arabic-English HSMT grammar is extended with 
an additional non-terminal symbol XO that can only generate fully lexicalized phrases, 
thereby disallowing recursive nesting of hierarchical rules (shallow-1 grammar). To 
account for the movement of large word blocks, other new non-terminals M^ allow for 
the monotonic generation of k non-terminals XO. While defining a much smaller search 
space than the original HSMT grammar, the resulting shallow grammar can capture 
the long-range reordering of our running example even in the likely absence of a rule 
covering the whole subject span (that is rule 2 in Figure]^. 


In a related work specifically addressing the issue of long-range reordering, Braune, 


Gojun, and Fraser (2012 1 propose to relax the span constraint only for specific t 5 q)es of 


hierarchical rules which are more likely to capture long reordering patterns in German- 
English. For instance, rules whose source side starts with at least one terminal followed 
by one non-terminal and ends with at least one terminal (t+ X t+) can capture the pattern 
'finite-auxiliary-verb X participle' (e. g. ist X gestiegen/has increased X) with very wide X 
spans. 


Mylonakis and Sima'an (2010) separate the modeling of local reordering (captured 


by fully lexicalized phrase-pair emission rules) from the modeling of higher-order 
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recursive reordering (captured by ITG-style non-lexicalized binary rules). Instead of a 
single non-terminal X, three different reordering-based labels are used according to the 
reordering pattern in which they participate: X for monofonic rules; XSL and XSR for fhe 
firsf and second symbol, respecfively, of swapping rules. Thus reordering decisions are 
conditioned on the phrase pair's content, rather than its lexical context as in HSMT. 
More fine-grained non-ferminals are infroduced by Mailleffe de Buy Wenniger and 
Sima'an (2014| fo also capfure fhe relafion of a phrase pair's reordering with respect 
to the parent phrase that contains it. 

Rather than relabeling non-terminals, other work incorporates reordering-specific 
models as additional feature functions. He, Meng, and Yu (2010[ | add to their HSMT 
grammar the generic phrase swapping rule (x —>■ X 1 X 2 IIX 2 X 1 ) and use a maximum- 
entropy classifier fo predicf whefher two neighboring phrases should be swapped or 
nof during decoding. Rafher fhan condifioning fhe decision on fhe whole phrase pair, 
fhe classifier employs feafures exfracfed from if, such as firsf and lasf word (or POS fag) 
of fhe source and fargef side. A similar model was firsf developed by Xiong, Liu, and 
Lin (2006) for simpler phrase franslafion models (i. e. wifhouf disconfinuifies) based on 
ITG. Li, Liu, and Sun (2013| use recursive aufoencoders (Socher ef al. 2011|l fo assign 
vecfor represenfations fo the neighboring phrases given as input to the ITG classifier, 
thereby avoiding manual feature engineering but affecting h 5 rpothesis recombination 
and decoding speed. Nguyen and Vogel (2013 1 and Huck et al. (2013|l successfully 
infegrafe fhe disforfion cosf feafure function and phrase orienfafion models initially 
designed for sfring-based PSMT info a charf-based HSMT decoder. 

Finally, Sefiawan, Kan, and Li (2007| observe fhaf, in languages like Chinese and En¬ 
glish, function words provide imporfanf clues on fhe grammatical relationships among 
phrases. Consequenfly fhey infroduce a SCFG where function words (approximated by 
high-frequency words) are fhe only lexicalized non-ferminals guiding phrase reorder¬ 
ing. Based on the same intuition, Setiawan et al. (2009} augment a HSMT system with a 
function-word ordering model that predicts, for any pair of franslation rules, which one 
should dominafe fhe ofher in fhe hierarchical sfrucfure, based on fhe function words 
thaf fhey contain]^ 


2.3.3 Tree-based SMT with soft syntactic constraints. We have discussed SMT frame¬ 
works where fhe translation model is fully based on fhe S 5 mfacfic parse free of fhe source 
or fargef sentence (Section 2.3. 1| or where S 5 mtax is not used at all (Section 2.3.2[ . A third 
line of work bridges between fhese fwo by exploiting S 5 mfacfic information in fhe form 
of soff consfrainfs while operating wifh a synchronous franslafion grammar exfracfed 
from non-parsed parallel dafa. 

Chiang (2005) firsf experimenfed wifh a feafure funcfion rewarding franslafion rules 


applied fo full S 5 mfacfic consfifuenfs (constituent feature). While this initial attempt did 
not appear to improve translation quality, Marton and Resnik (2008| further elaborated 
the idea and proposed a series of finer-grained feafures distinguishing among con- 
sfifuenf fypes (VP, NP, efc.), evenfually leading fo better performance. Gao, Koehn, and 
Birch (2011| exfracf fwo reordering-relafed feafure funcfions from source dependency 
parse frees: (i) The dependency orientation model predicts whether the relative order 
of a source word and ifs head should be reversed during translation. This is trained as 


9 Two other models ut ilizin g function words as the anchors of global reordering decisions are proposed in 
jSetiawan e t al. 201 31 and (Setiaw a n, Zh ou, and Xiang 20131. Although integrated in a syntax-based 
system jbhen, Xu, and Weischedel 2Uiu|, these models are in principle applicable to other SMT 
frameworks such as JTSM t . 
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a maximum-entropy classifier using the words and their dependency relation type as 
features, (ii) The dependency cohesion penalty fires whenever a word and its head are 
translated separately (i. e. by different translation rules) thereby measuring derivation 
well-formedness. Since long-range reordering tends to happen closer to the root and 
local reordering closer to the leaves, a distinction is made between words occurring at 
different depths of the dependency tree leading to a number of sub-features. In this 
way, the tuning process can decide how important or reliable are feature scores coming 
from different levels of the parse tree. Huang, Devlin, and Zbib (2013| worked instead 
with constituency parses and trained a classifier to predict whether the order of any 
two sibling constituents in the input tree should be reversed or maintained during 
translation. The classifier is trained by maximum-entropy using a number of S 5 mtactic 
features and used during decoding at the word level: that is, each pair of input words 
inherit the orientation probabilities of the constituents that cover them respectively. 

Syntactic armotation has been also used to refine non-terminal SCFG labels, poten¬ 
tially leading to better reordering choices. In (Zollmann and Venugopal 2006) and (|My- 
lonakis and Sima'an 201 1|, labels indicate whether a phrase corresponds to a S 5 mtactic 
constituent or to part of it, as well as the constituent type, relatively to a target or source 
parse tree, respectively. Moreover, Mylonakis and Sima'an (2011 1 treat the phrase-pair 
category as a latent variable and let their system learn reordering distributions over 
multiple labels per span (e.g. generic X or source-S 5 mtax based like NP, VBZ+DT, etc.). 
Li et al. (2012| use source dependency annotation to refine non-terminal symbols with 
S 5 mtactic head information. More specifically, given a hierarchical phrase, its type is 
obtained by concatenating the POS tags of the exposed heads it contains on the source 
side, where an exposed head is a word dominated by a word outside the phrase. Like 
He, Meng, and Yu (2010) , Li et al. (2012) also allow adjacent phrases to swap but instead 
of introducing a separate orientation model, they rely on rule translation probabilities 
based on the refined non-terminals to guide reordering. 


2.4 Word Reordering as Pre- (or Post-) Processing 


Given the complexity of solving word reordering during the decoding process, a pro¬ 
ductive line of research has focused on decoupling reordering decisions from translation 
decisions. These approaches aim at arranging words in a target-like order either on the 
input, before translating, or on the output, after translating. Thus, word reordering is 
solved as pre- or post-processing (i. e. pre-ordering or post-ordering) in a monolingual 
fashion and with unconstrained access to the whole sentence context. Figure |7 (Sudoh 
et al. 2011} illustrates the workflows of pre- and post-ordering approaches as opposed 
to standard SMT. 


2.4.1 Main pre-ordering strategies. A large number of pre-ordering strategies has been 
proposed. As a first classification, we divide them into deterministic, non-deterministic 
and hybrid. Deterministic pre-ordering aims at finding a single optimal permutation 
of the input sentence, which is then translated monotonically or with a low distortion 
limit ([Niefien and Ney 2(ioT Xia and McCord 2004 jCollins, Koehn, and Kucerova 2005 


Habash 20071|Li et al. 2607||Tromble and Eisner 2009) Xu et al. 2009) Genze 

2010) Isozaki 

et al. 2010b) 

Yeniterzi and Oflazer 2011 

1) Khalilov and Fonollosa 2011] 

Khalilov and 

Sima'an 2011 

Visweswariah et al. 2011 

Cojun and Fraser 2012 Yang et a 
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(cl) 



Figure 7: Typical workflows of standard, pre-ordering and post-ordering approaches to 
SMT. Taken from Sudoh et al. (2011 1 . 


and Petrov 2013| Jehl et al. 2014| P^ Non-deterministic pre-ordering encodes multiple 
alternative reorderings into a word lattice and lets a monotonic (usually n-gram based) 
decoder choose the best path accordi ng to its models (|Zens, Och, a nd Ney 2002 Kanthak 
et al. 2005} Crego and Marino 2006 Zhang, Zens, and Ney 200^ Rottmann and Vogel 
2007 Crego and Habash 2008||Elm ing and H abash 2009) [Niehues and Kolss 2009|. A 
hybrid approach is adopted by [Bisazza and Federico (2010| , [Andreas, Habash, ana 
Rambow (2011) : rules are used to generate multiple likely pre-orderings, but only for 
specific language phenomena that are responsible for difficult (long-range) reordering 
patterns. The sparse reordering lattices produced by these techniques are then trans¬ 
lated by a decoder performing additional phrase-based reordering. In a follow-up work, 
Bisazza and Federico (2012| introduce another way to encode multiple pre-orderings 
of the input: instead of generating a word lattice, pre-computed permutations are 
represented by a modified distortion matrix so that lower distortion costs or 'shortcuts' 
are permitted between selected pairs of input positions. 

Pre-ordering methods can also be classified by the kind of pre-ordering rules that 
they apply: that is, manually written based on linguistic knowledge, or automatically 
learned from data. We now discuss each of them in detail. 

2.4.2 Linguistic knowledge based pre-ordering. In these approaches, manually written 


rules determine the transformation of in 

out syntax trees (|Collins, Koehn, and Kucerova 

2005 |Wang, Collins, and Koehn 2007 

Xu et al. 2009 Isozaki et al. 2010bj Yeniterzi 


and Oflazer 2010{ |Gojun and Fraser 2012 Andreas, Habash, and Rambow 2011 1 or 


the permutation of shallow S 5 mtactic chunks in a sentence (Hardmeier, Bisazza, and 


Federico 2010 |Durgar El-Kahlout and Oflazer 2010 Bisazza, Pighrn, and Federico 2012 1 . 

In an early example of S 3 mtax-based pre-ordering, |Collms, Koehn, and Kucerova (2005^ 

propose a set of six rules aimed at arranging German sentences in English-like order. 
The rules address the position of verbs, verb particles and negation particles, and they 
are applied to constituency parse trees. Eollowing a similar approach, Gojun and Fraser 
(2012) develop a set of rules for the opposite translation direction (English-to-German). 


10 |Li et al. (2007) experiment with a small number of n-best pre-orderings given as alternative inputs to the 
bMl system. 
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Xu et al. (2009[ instead propose a simple set of dependency-based rules to pre-order 
English for translation into subject-object-verb (SOV) languages, which is shown to be 
effective for Korean, Japanese, Hindi, Urdu, and Turkish. Isozaki et al. (2010b| obtain 
even better results in an English-to-Japanese task using onl 


y one pre-ordering rule, i. e. 


head finalization, with a parser annotating syntactic heads. 


2.4.3 Data-driven pre-ordering. This kind of models are learned from sets of pairs (f, f') 
where f is a source sentence and f' is its reference permutation (pre-ordering) inferred 
from a reference translation e via a word-level alignmentj^These approaches typically 
require some form of linguistic annotation of the source language, such as syntactic 


Genzel 2010j Khalilov and Eonollosa 2011| 

Khalilov and Sima'an 2011||Yang et al. 2012 

Lerner and Petrov 2013| 

Jehl et al. 2014|, shallow S 5 mtax chunks 1 Zhang, Zens, and Ney 

2007 Crego and Habash 2008| or POS labels (|Crego and Marino 2006 Rottmann and 

Vogel 2007 Niehues and Kolss 2009 Tromble and Eisner 2009 Visweswariahetal. 2011||. 


Among the first examples of data-driven tree-based pre-ordering, Xia and McCord 
(2004^ propose a method to automatically learn reordering patterns from a dependency- 
parsed Erench-English bitext, using a number of heuristics. While source-side parses are 
required by their method, target-side parses are optionally used to provide additional 
constraints during rule extraction. [Habash (2007[ | extracts pre-ordering rules from an 
Arabic-English parallel corpus dependency-parsed on the source side. In both these 
works, pre-ordering rules are applied in a deterministic way to preprocess both training 
and test data. Eollowing a discriminative modeling approach, |Li et al. (2007| train a 
maximum-entropy classifier to pre-order each node with at most 3 children in the source 
constituency parse, using a rich set of lexical and S 5 mtactic features. Lerner and Petrov 
(2013[ extend this work to pre-order nodes with more children (up to 7 on either side 
of the head) using a cascade of classifiers: first, decide the order of each child relative 
to the head, then decide the order of left children and that of the right children. As 
training separate classifiers for each number of children is prone to sparsity issues, 
Jehl et al. (20141 build a single logistic regression model to predict whether any two 


sibling nodes should be swapped or not. Then, for each node in the tree, they search 
for the best permutation of all its children given the pairwise scores produced by 
the model, using a depth-first procedure. Yang et al. (2012) treat the permutation of 
each node's children as a ranking problem and model it with ranking support vector 
machines. As an alternative to deterministic pre-ordering, they also propose to use the 
predicted source permutation to generate soft constraints for the SMT decoder: that is, 
a penalty that fires whenever the decoder violates the predicted pre-ordering. A tighter 
integration between source pre-ordering and source-to-target translation is proposed 
by Dyer and Resnik (2010| . In their approach, optimal source pre-orderings (f') are 
treated as a latent variable in an end-to-end translation model and the parameters of the 
tree permutation model are learnt directly from parallel data. At test time, alternative 
permutations of the input tree are encoded as a source reordering forest, which is then 
translated by a finite-state phrase-based translation model. 


Examples of pre-ordering based on shallow S 5 mtax include (Zhang, Zens, and Ney 
200 7| and ([Crego and Habash 2008|. In these approaches, automatically extracted chunk 


11 Various heuristics have been proposed to convert a word alignment set into a sentence perm utation 
(Birch, Osborne, and Blunsorn 2010||Feng, Mauser, and Ney 2010[[Visweswariah et al. 2011|. 
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pre-ordering rules are used to generate a word reordering lattice of the input sentence, 
which is then translated by a monotonic phrase or n-gram based decoder. 

In ( |Costa-jussa and Fonollosa 2006) , pre-ordering is learnt by training a monolin¬ 
gual n-gram based SMT system at the level of word clusfers. In ( [Tromble and Eisner 
2009|l, pre-ordering is casf as a permufafion problem and solved by a model thaf es- 
fimafes fhe probabilify of reversing fhe relafive order of any two inpuf words based 
on fheir disfance as well as lexicalized and POS-based feafures. In a relafed work, 
Visweswariah ef al. (20H| obfain smaller models and better resulfs by learning fhe cosf 
of a given inpuf word appearing right after another, as opposed to an 5 rwhere after it (cf. 
source word-after-word reordering models described in Section[2T). 


2.4.4 On the limitations of syntax-based pre-ordering. S 5 mtax is often regarded as the 
most effective way to inform reordering in franslafion. However, empirical work has 
shown fhaf fhe success of S5mfax-based pre-ordering mefhods can be severely limbed 
by: (i) fhe reachabilify of reference permufafions when parse frees are used fo consfrain 
fhe pre-ordering model, and (ii) fhe qualify of fhe parser used fo learn and apply a pre¬ 
ordering model. 


Wifh regard to the constraints imposed by S 5 mtactic trees (i), Khalilov and Sima'an 


(2012| conducted oracle pre-ordering experiments across various language pairs. Their 


results consistently showed that final franslafion qualify was highesf by far when no 
synfacfic consfraint was imposed on pre-ordering (oracle sfring). On fhe confrary only 
allowing permufations of siblings of fhe source parse free (oracle free) gave fhe smallesf 
improvemenf. Only some of fhis loss could be recovered by applying specific modifica¬ 
tions fo fhe free before exfracfing fhe opfimal permufafion (oracle modified free). 

Wifh regard fo parser accuracy (ii), [Green, Sathi, and Manning (20091 analyzed two 


sfafe-of-fhe-arf parsers (Bikel 2004 


Klein and Manning 2003) and reporfed F-measures 


of only 55-56% af fhe sub-fask of defecting Arabic NP subjecfs in verb-inifial clauses. 
Similar resulfs were observed by Carpuaf, Marfon, and Habash (2010) using a depen¬ 
dency parser (|Nivre, Hall, and Nilsson 2006 '. The same paper also showed fhaf fhe 
correcf pre-ordering for Arabic-English franslafion could nof be safely predicfed even 
from gold sfandard parses, parfly due fo S 5 mfacfic fransformafions occurring during 


franslafion. From a manual analysis of fheir English-German sysfem, Gojun and Eraser 
(2012) reporfed fhaf abouf 10% of fhe English clauses were wrongly pre-ordered, mosfly 


due fo source senfence parsing errors. Howleff and Dras (2011 1 analyzed a reimple- 


menfafion of fhe German pre-ordering method of [Gollins, Koehn, and Kucerova (2005) 
and found fhaf resulfs could be affecfed — or even cancelled ouf — by many facfors 
including: choice of framing dafa, qualify of fhe parser, as well as order of fhe fargef 
language model and t 5 rpe of reordering model used during decoding. 

Rafher fhan relying on supervised parsers framed on golden freebanks, specific 


parsers can be induced direcfly from non-annofafed parallel fexf. In (DeNero and 
Uszkoreif 2011), source senfence reorderings are tirsf inferred from fhe word alignmenf 


with the target translation. Then, a binary parsing model is trained to maximize the 
likelihood of source frees fhaf can generate such reorderings. Einally, a pre-ordering 
model is trained to permute each node in the tree. Evaluated on the English-Japanese 
language pair, this method almost equals the performance of a pre-ordering mefhod 
based on a supervised parser. Neubig, Wafanabe, and Mori (2012) follow a similar 


approach buf build a single ITG-sfyle pre-ordering model freafing fhe parse free as a 
lafenf variable. In fhe fargef self-fraining mefhod of Kafz-Brown ef al. (2011) , a baseline 
freebank-frained parser is used fo produce n-besf parses of a parallel corpus' source 
side. Then, the parses resulting in the most accurate pre-ordering after application of a 


18 























































Bisazza and Federico 


A Survey of Word Reordering in Statistical Machine Translation 


dependency-based pre-ordering rule set ( |Xu et al. 2009[ l are added to the treebank data 
and used to re-train the baseline parser. 


2.4.5 Post-ordering. A somewhat smaller line of research has instead treated reordering 
as post-processing. In ( Bangalore and Riccardi 2000{ Sudoh et al. 2011) , target words are 
reordered after a monotonic translation process. Other work has focused on rescoring a 
sef of n-besf franslafion candidafes produced by a regular PSMT decoder — for insfance 
by means of POS-based reordering femplafes (|Chen, Ceffolo, and Federico 2006) or 
word-class specific distortion models (|Gupta, Cettolo, and Federico 2007^Chang and 


Toutanova (2007) use a dependency tree reordering model to generate n alternative 


orders for each 1-besf senfence produced by the SMT system. Each set of n senfence 
reorderings is fhen reranked using a discriminative model framed on word bigram 
feafures and sfandard word reordering feafures (i. e. disfance or orienfation between 
consecufively franslafed inpuf words). 

Focusing on Japanese-fo-English franslafion, Sudoh ef al. (2011 2013| proposed to 
Translate' foreign-order English info correcf-order English using a monolingual phrase- 
based ( Sudoh ef al. 20lT| | or S 5 mfax-based (Sudoh ef al. 20131 SMT sysfem framed for 
fhis specific subfask|^^|The underlying mofivafion is fhaf, while English-fo-Japanese is 
well handled by pre-ordering wifh fhe aforemenfioned head-finalization rule ([Isozaki 


|et al. 2010b|, it is much harder to predict the English-like order of Japanese consfifuenfs 
for Japanese-fo-English translation. Post-ordering addresses this issue by generating 
head-final English (HFE) senfences fhaf are used to create a HFE-fo-English parallel 
corpus. Gofo, Ufiyama, and Sumifa (201^ 2013| solve posf-ordering by parsing fhe 
HFE senfences info binary frees armofafed wifh bofh S 5 mfacfic labels and ITG-sfyle 
monofone/swap labels. Hayashi ef al. (2013| improve upon fhis work wifh a shiff- 
reduce parser fhaf efficienfly infegrafes non-local feafures like n-grams of fhe posf- 
ordered sfring. 

Also related fo posf-ordering is fhe work on righf-fo-leff or reverse decoding by 
Wafanabe and Sumifa (2002) , Finch and Sumifa (2009[ l, and Freifag ef al. (2013) . Here, 


fhe fargef senfence is builf up from fhe lasf word fo fhe firsf, fhereby altering language 
model confexf and reordering search space. Finch and Sumifa (2009} obfain besf resulfs 
on a wide range of language pairs by combining fhe outpufs of sfandard and reverse 
decoding systems. 


3. Evaluating Word Reordering in Statistical Machine Translation 

Since there are irmumerable ways to correctly render a source sentence's meaning in 
the target language, automatically evaluating translation quality is a complex problem. 
Generally, SMT systems are judged by the extent to which their outputs resemble a 
set of reference franslafions produced by differenf human franslafors. Despite relying 
on a very rough approximation of language variabilify, fhis approach provides SMT 
researchers wifh fasf aufomafic mefrics fhaf can guide, af leasf in parf, fheir sfeps 
towards improvemenf. Besides, fasf evaluation mefrics are used fo aufomafically fune 
SMT feafure weighfs on a developmenf corpus, for insfance by means of minimum error 
rate training procedures ( |Och 20(13) . The design of MT evaluafion mefrics correlafing 
with human judgements is an active research area. Here we briefly survey fwo widely 


12 Note the similarity to the pre-ordering approach of|Costa-jussa and Fonollosa (2006|, except that here the 
monolingual SMT process is applied to the target language atter a monotomc translation phase. 
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used general-purpose metrics, BLEU and METEOR, and then describe in more detail a 
number of reordering-specific mefrics. 


3.1 General-purpose metrics 


BLEU i Papineni et al. 2001[ is a lexical match based score that represents the de facto 
standard for SMT evaluation. Here, proximify between candidafe and reference fransla- 
fions is measured in ferms of overlapping word n-grams, wifh n f 5 rpically ranging from 
1 fo 4. Eor each order n a modified precision score (see Papineni ef al. (2001) for defails) 
is compufed on fhe whole fesf sef and combined in a geomefric mean. The resulting 
score is fhen multiplied by a brevity penalty fhaf accounfs for lengfh mismafches between 
reference and candidafe franslafions. Al-Onaizan and Papineni (2006i use BLEU fo 


measure word order similarity between two languages: that is, by computing the BLEU 
score between the original target sentence e and a source-like permutation of e. Using 
n-grams, fhough, is a limifed solufion to the problem of word ordering evaluation: 
Eirsf, because only exacf surface mafches are counfed, wifhouf any considerafion of 
morphology or S 5 monymy. Second, because fhe absolufe posifioning of words in fhe 
senfence is nof capfured, buf only fheir proximify wifhin a small confexf. 


The former issue is addressed fo some exfenf by METEOR (Banerjee and Lavie 


20051, which relies on language-specific sfemmers and synonymy modules fo go beyond 


fhe surface-level similarify. As for word order, METEOR freafs if separafely wifh a 
fragmentation penalty proporfional fo fhe smallesf number of chunks fhaf fhe h 5 rpofhesis 
musf be divided into to align with the reference translafion. This quantify can be 
inferprefed as fhe number of times fhat a human reader would have fo 'jump' between 
words fo recover fhe correcf translation order. However, no distinction is made between 
short and long-range reordering errors. 

The weakness of BLEU and METEOR wifh respecf fo word order was demonsfrafed 
by Birch, Osborne, and Blunsom (2010| wifh a significanf example fhaf we reporf in 
Table For simplicify, fhe example assumes fhaf fhe reference order is monofonic and 
fhaf h 5 rpofheses and reference franslafions confain exacfly fhe same words. According 
fo bofh mefrics, h 5 q)ofhesis (a) is worse fhan (b), alfhough in (a) only two adjacenf words 
are swapped while in (b) fhe two halves of fhe senfence are swapped. 


3.2 Reordering-specific metrics 


To overcome the aformentioned limitations, Birch, Osborne, and Blunsom (2010 1 pro 


pose to directly measure the similarity between the reorderings needed to reach the 
reference franslations from fhe source senfence and fhose applied by fhe decoder fo 
produce fhe candidafe franslafion. In practice, fhis is done by firsf converting word 
alignmenfs fo permufations using simple heuristics fo handle null and multiple align- 
menfs, and fhen computing a permufafion disfance among fhe resulfing permufafions. 
Among various mefrics proposed in fhe paper, fhe square roof of fhe Kendall's Tau was 
shown fo be reliable and highly correlafed wifh human judgemenfs. 

The normalized Kendall's Tau disfance K is originally a measure of disagreemenf 
befween rankings. Given a sef of n elemenfs and fwo permufafions tt and a, fhe K 
disfance corresponds fo fhe number of discordanf pairs (i.e. pairs of elemenfs whose 
relative order differs in fhe fwo permufations) normalized by the total number of 
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i-WCO^lOtOh-COWT- 



(1 234»6»5»789 10) 
Example (a) 


i-Meo^intor-coo»i- 



(6 7 8 9 10 • 1 2 3 4 5) 
Example (b) 


Example: 

(a) 

(b) 

BLEU 

61.80 

81.33 

METEOR 

86.91 

92.63 


Table 2: Two example alignments and their respective BLEU and METEOR scores, 
assuming that the reference alignment is monotonic. The permutation resulting from 
fhe hypofhesis alignmenf is reporfed under each mafrix, where bullef poinfs represenf 
jumps befween non-sequenfial indices. Taken from Birch (2011 1 . 


ordered elemenf pairs: 


K{tt, a) 


^n{n — 1 ) 


where d{i,j) 


1 if TTi < TTj and Ui > aj 
0 ofherwise 


Birch, Osborne, and Blunsom (2010) further suggest to extract the square root of K fo ob- 
fain a function thaf is more discriminafive on lower disfance ranges, i.e. for franslafions 
fhaf are closer fo fhe reference word ordering. Einally, fhe Kendall Reordering Score 
(KRS) — a positive measure of qualify ranging from 0 fo 1 — is compufed by subfracfing 
fhe laffer quanfify from one, and by multiplying fhe resulf by a brevify penalfy (BP) fhaf 
accounfs for lengfh mismafches befween reference and candidafe franslafions: 


KRS{tt, cr) = (1 — yjK{' k, a)) ■ BP 


The BP definifion corresponds fo fhaf of BLEU ( Papineni ef al. 2001} wifh fhe difference 
fhaf, for KRS, if is compufed af fhe senfence level. In case of mulfiple references, fhe one 
fhaf yields fhe highesf score for each fesf senfence is refained. Einally, fhe average of all 
senfence-level KRS scores gives fhe global KRS of fhe fesf sef. The linear interpolation 
of KRS and BLEU (LRscore) can be successfully used fo optimize fhe feafure weighfs of 
a PSMT sysfem, leading fo franslation oufpufs fhat are preferred by human annofafors 
according to Birch and Osborne (2011|. 

In a related work, |Bisa'z^a and Pederico (2013a| | observe that some word classes, like 
verbs, are t 5 q)ically more important than others to determine the general structure of a 
senfence. Hence, fhey develop a word-weighfed KRS varianf fhaf is more sensifive fo 
fhe positioning of specific inpuf words. Assuming fhaf each inpuf word fi is assigned a 
weighf Xi, fhe original KRS formula is modified as follows: 




Xi+Xj if TTi < TTj and ai > Uj 
0 ofherwise 
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For their evaluation of long reordering errors in Arabic-English and German-English, 
Bisazza and Eederico (2013a[l set the weights to 1 for verbs and 0 for all ofher words 


fo only capfure verb reordering errors. The resulfing mefric, KRS-V, rafes a franslafion 
hypofhesis as perfecf when fhe franslafions of all source verbs are locafed in fheir correcf 
position, regardless of fhe ordering of ofher words. 

In a differenf approach called RIBES, Isozaki ef al. (2010a) propose fo direcfly 
measure fhe reordering occurring between the words of fhe h 5 rpofhesis and fhose of 
the reference translafion, fhereby eliminafing fhe need fo word-align inpuf and outpuf 
senfence. A limifafion of fhis approach is fhat only identical words contribute to the 
score. As a solution, the permutation distance is multiplied by a word precision score 
that penalizes hypotheses containing few reference words. Neverfheless, fhe resulfing 
mefric assigns differenf scores fo hypofheses fhaf differ in fheir lexical choice, buf nof in 
fheir word reordering. 

Talbof ef al. (2011) infroduce yef anofher reordering-specific mefric, called fuzzy 


reordering score (FRS) which, like fhe KRS, is independenf from lexical choice and 
measures fhe similarify befween a senfence's reference reordering and fhe reordering 


produced by an SMT sysfem (or by a pre-ordering technique). However, while Birch, 
Osborne, and Blunsom (2010| employed the Kendall's Tau between the two sentence 


permutations, Talbot et al. (201l[ | count the smallest number of chunks fhat the hypoth¬ 
esis permutation must be divided into to align with the reference permufation. This 
corresponds precisely fo fhe fragmenfafion penalfy of METEOR excepf fhaf fhe align- 
menf is performed befween permufafions and nof befween franslafions. Like METEOR, 
ERS makes no difference befween shorf and long-range reordering errors (cf. Table |^. 

[Sfanojevic and Sima'an (2014b) argue for a hierarchical freafmenf of reordering 
evaluation, where word sequences can be grouped recursively info larger blocks. To 
fhis end, fhey facforize fhe outpuf-reference reordering info a Permufafion Tree ([Zhang 


and Gildea 2007|, whose nodes represenf afomic permufafions. Given fhis factorization. 


fhe counfs of monofone (1 2) versus ofher permufation nodes — (2 1), (314 2), etc. 
are used as features in a linear model of translafion qualify (BEER) framed fo correlafe 
with the human ranking of a sef of MT sysfem outpufs. Wifh reference fo Table fhe 
permufafion frees of bofh h 5 rpofheses (a) and (b) would confain only one swapping 
node leading fo fhe same reordering score. Sfanojevic and Sima'an (2014a) extend fhis 
work wifh a sfand-alone reordering mefric fhaf considers all possible free facforizafions 
of a permufafion (permufafion foresf) and fhaf gives recursively less imporfance fo 
lower nodes in fhe free (i.e. covering smaller spans). Hierarchical permufafion mefrics 
are shown fo beffer correlafe wifh human judgemenfs fhan sfring-based permufafion 
mefrics like fhe Kendall's Tau disfance K. 


4. Reordering Phenomena in Natural Languages 

Understanding the complexity of reordering in a given language pair is key fo selecting 
the right SMT models and to improving them. To date, word reordering phenomena in 


amount of reordering is cerfarnly imporfanf, undersfanding which kinds of reordering 
occur in a given language pair is also essential. To fhis end, we presenf a qualifafive 
analysis of word reordering based on linguisfic knowledge. More specifically, we draw 
on a large body of S 5 mfacfic information collected by linguists from more fhan 1500 


nafural languages have mainly been analyzed from a quanfifafive perspective (Birch, 


Osborne, and Koehn 2008 Birch, Blunsom, and Osborne 2009 j. While measuring the 
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languages, and systematized in the World Atlas of Language Structures (WALS) ( [Dryer 
and Haspelmath 201 Ijp^ 

Following the seminal work of language fypologisf Matthew S. Dryer, we describe 
fhe word order profile of a language by fhe canonical orders of ifs consfifuenf sefs (word 
order feafures). The resulfing language pair classification is primarily based on fhe order 
of subjecf, objecf and verb, and furfher refined according fo fhe order of several ofher 
elemenf pairs, such as noun-adjecfive, verb-negafion, efc. We fhen compare fhe word 
order feafures of several languages fhaf were sfudied in fhe SMT field, and show fhaf 
empirical resulfs generally confirm fhe existing fheorefical knowledge. 


4.1 A qualitative analysis 


The amount of word reordering found in a language pair is known fo be a good predic¬ 
tor of SMT performance. Birch, Osborne, and Koehn (2008 1 considered fhree variables — 


reordering quantify morphological complexify and hisforical relafedness — and found 
fhe firsf fo have fhe highesf correlafion wifh fhe BLEU scores of a sfandard PSMT system 
on a sample of 110 European language pairs. Birch, Blunsom, and Osborne (2009) 
furfher analyzed fhe disfribufion of difterenf reordering widfhs in Arabic-English and 
Chinese-English, and fhe abilify of two SMT approaches fo model fhem. They found 
fhaf fhe PSMT approach is more suifable for language pairs where mosf reordering 
is local (Arabic-English), while fhe hierarchical approach is sfronger when medium- 
range reorderings are dominanf (Chinese-English). Still, bofh PSMT and HSMT failed 
fo capfure mosf of fhe long-range reorderings found in fhe reference corpora. 

These findings are indeed relevanf fo our work, buf we believe fhere is also much 
fo learn from fheorefical linguistic knowledge. Moreover, a quanfifafive analysis can 
suffer from noise in fhe dafa, fypically originating from aufomafic word alignmenfs. 
Birch, Blunsom, and Osborne (2009) used manual word alignmenf in fheir sfudy, buf 
fhis kind of resource is available only for very few language pairs. Noise can also be due 
fo whaf we can call optional reordering: human franslafors otten choose fo resfrucfure 
fhe sentence according fo genre conventions or fo fheir personal sfyle, even when fhis is 
nof required by fhe fargef language grammar. Here is an example: 


Arabic sentence: 

S’ 

^ ^ i ... I 4j 1 ^ j 1 I 1 1 Hi 11 ^ 55 ^ 

. “ 5 “ ” JU 

Literal translation: 

Bush, aged 55, assured journalists before leaving the White House that he felt 
“great” and that his health was “very good”. 

Human translation: 

Before leaving the White House, Bush, aged 55, assured journalists that he 
felt “great” and that his health was “very good”. 


As also noted by Pox (200^ , this kind of reordering is not strictly necessary to produce 
accurate and fluent translations, but its occurrence in parallel corpora affects the auto¬ 
matic reordering measures. 


13 http://wals.info 
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On the contrary, a qualitative analysis can profit from fhe exfensive work done by 
linguisfs and grammaficians fo absfracf fhe fundamenfal properties of a language. In 
fhis section we draw largely on Dryer (2007[ | and on fhe sections of WALS devofed fo 
word order ( Dryer (2011[ |, ch. 81-97,143-144). 


4.2 Word order profiles 


The word order profile of a language is defermined by fhe canonical order of ifs 
consfifuenf sefs, or word order feafures. In general, fhe basic or canonical order of a con- 
sfifuenf sef can be esfablished by criferia of frequency (fhe mosf common), disfribufion 
(fhe one with the least restricted usage) or pragmatics (the neutral one) (Dryer 2007|. 
Although some languages are said to have free (or flexible) order, if is offen possible 
fo defecf one fhaf is dominanf and neufral. Consider for insfance English, a subjecf- 
verb-objecf (SVO) language where ofher orders are used, but only to achieve specific 
emphasis or fopicalizafion effecfs: 

(1) a. I saw the cat. 
b. The cat, I saw. 

However, there exist cases where no particular order can be defined as dominant. An 
example of mix-ordered constituent set in English is the pair noun and genitive: 

(2) a. the tail of the cat 
b. the cat’s tail 


we 


Based on Dryer (2007| and on the availability of data points in the WALS, 
have established a set of 13 core features to determine the word order profile of a 
language. Eor the purpose of describing word order differences between language pairs, 
we have divided the features into two broad categories: clause-level and phrase-levef^ 
An English example for each feature is provided in Table 


4.2.1 Clause-level order features. 


Subject, Object, Verb [WALS feature 81A] 

The first and most important feature is the "ordering of subject, object, and 
verb in a transitive clause, more specifically declarative clauses in which 
both the subject and object involve a noun (and not just a pronoun)" ( [Dryer 
2011) . Eor instance, English and Erench are SVO languages, while Turkish is 
SOV. The distribution of main word order t5q3es in a large sample of world 
languages is given in Table This feature is often used alone to denote 
the word order profile of a language, because it can be a good predictor of 
several other features. 

Oblique or Adpositional Phrase [84A] 

This feature refers to the position of a phrase functioning as adverbial 
modifier of the verb, relative to the position of object and verb. Eor instance, 
English is VOX because it places oblique phrases after verb and object. 


14 In this section, phrase is used in its traditional syntactic sense — i. e. a group of words forming a 
constituent — as opposed to the notion of data-driven phrase adopted by phrase-based SMT. 
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Order 

Languages 

sov 

565 

41% 

svo 

488 

35% 

vso 

95 

7% 

VOS 

25 

2% 

ovs 

11 

1% 

osv 

4 

<1% 

mixed / no-dominant 

189 

14% 

total sample size 

1377 


Table 3: The distribution of main word order types (Subject, Object, Verb) in the world 
languages. From the World Atlas of Language Sfrucfures, chapfer 81 (Dryer 2011). 


• Noun and Relative Clause [90A] 

Order of fhe relative clause wifh respecf fo fhe noun if modifies. 

• Adverbial Subordinator and Subordinate Clause [94A] 

Subordinators are used to link adverbial subordinate clauses to the main 
clause. They can take the form of verbal suffixes or separafe words, such as 
the English subordinating conjunctions 'when' and 'because'. 

• Polar Question Particle [92A] 

In many languages, polar (yes/no) questions are signaled by specific par- 
ficles. This feafure denofes fheir position in fhe senfence (nof defined for 
English). 

• Content Question Phrase [93A] 

Content questions are characterized by the presence of an inferrogafive 
word or phrase (e.g. 'who', 'which one'). In some languages, like English, 
fhese are always placed af fhe beginning of fhe senfence. In some ofhers, 
like Turkish, fhey fake fhe posifion of fhe consfifuenf fhey replace: for 
insfance, fhe word 'ne/whaf' replacing fhe objecf nafurally occurs befween 
subjecf and verb. 

• Negation and Verb [143A] 

Order of fhe negafive word or morphem^^ wifh respecf fo fhe main verb. 
Nofe fhat more fhan one word or morpheme may be necessary fo express 
negation (e. g. ‘ne... pas' in Erench). 

4.2.2 Phrase-level order features. 

• Noun and Adpositions [WALS feature 85A] 

Whether a language uses mainly prepositions or postpositions. 

• Noun and Genitive [86A] 

Order of genitive or possessor noun phrase wifh respecf fo fhe head noun. 

• Noun and Adjective [87A] 

Order of adjectives wifh respecf fo fhe noun fhey modify. 

• Noun and Demonstrative [88A] 


15 Unlike the WALS, we do not distinguish between negative words and affixes for this feature. 
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Order of demonstrative words (e.g. this, that) or affixes with respect to the 
noun they modify. 

Noun and Numeral [89A] 

Order of cardinal number words with respect to the noun they modify 
Adjective and Degree Word [91 A] 

Order of degree words (e.g. very, more) with respect to the adjective they 
modify. 




Indo-European Afro-Asiatic 

Germanic Romance Semitic 

Altaic 

Turkic 

Japanese 

Japanese 

Sino-Tibet. 

Chinese 


Features 

English German French 

Arabic 

Turkish Japanese Chinese 


Subject,Object,Verb 
[Tom] [chases] [Jerry] 

s-v-o 

S-V-O/ 

s-o-v 

S-V-O 

v-s-o/ 

S-v-o* 

S-O-V 

S-O-V 

S-V-O 


Oblique Phrase 
[chases] [Jerry] [with a stick] 

V-O-X 

mixed 

V-O-X 

V-O-X 

X-O-V 

X-O-V 

x-v-o 

01 

> 

01 

N oun,RelClause 

[a stick] [that he stole] 

N-Rel 

N-Rel 

N-Rel 

N-Rel* 

Rel-N 

Rel-N 

Rel-N 

1/1 

S 

u 

Subordinator,Clause 

[because] [he was hungry] 

Sub-C 

Sub-C 

Sub-C 

Sub-C 

C-Sub/ 

Sub-C 

C-Sub 

mixed** 

PolarQuest.Particle 

0 [did Tom steai it?] 

none 

none 

initial 

initial 

final 

final 

final 


ContentQuest.Phrase 

[what] [did Tom steal?] 

initial 

initial 

initial 

initial* 

other 

other 

other 


Negation, Verb 
he did [not] [steal] 

Neg-V 

Neg-V/ Neg-V-Neg/ 
V-Neg V-Neg 

Neg-V 

V-Neg 

V-Neg 

Neg-V 


N oun, Adpositions 
[with] [a stick] 

Adp-N 

Adp-N 

Adp-N 

Adp-N 

N-Adp 

N-Adp 

N-Adp/ 

Adp-N 

01 

Noun,Genitive 

[Tom's] [stick] 

N-Gen/ 

Gen-N 

N-Gen 

N-Gen 

N-Gen 

Gen-N 

Gen-N 

Gen-N 

> 

01 

1/1 

5 

XI 

EL, 

Noun,Adjective 
[hungry] [Tom] 

A-N 

A-N 

N-A 

N-A 

A-N 

A-N 

A-N 

N oun,Demonstrative 

[this] [stick] 

Dem-N 

Dem-N 

Dem-N 

Dem-N 

Dem-N 

Dem-N 

Dem-N 


Noun,Numeral 

[two] [sticks] 

Num-N 

Num-N 

Num-N 

Num-N 

Num-N 

Num-N 

Num-N 


Adjective,DegreeW. 
[very] [hungry] 

Deg-A 

Deg-A 

Deg-A 

A-Deg 

Deg-A 

Deg-A 

Deg-A 


Feature 

English German Erench 

Arabic 

Turkish Japanese Chinese 


Table 4: The word order profile of seven world languages. Language family and genus 
(Dryer 1989) are indicated in the header's first and second row, respectively. Sources: the 
World Atlas of Language Structures (Dryer and Haspelmath 2011), {*) authors' knowledge, 
and {**) (Li 2008). 
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4.2.3 Language sample. For our study, we have chosen seven widely spoken languages. 
These are English, German, French, Arabic (Modern Standard), Turkish, Japanese and 
Chinese (Mandarin). Mainly based on the WATS, we have summarized the word order 
feature values for all fhese languages in Table Whenever possible, feafures were 
assigned one (or fwo) values corresponding fo fhe dominanf order(s) in fhaf language. 
When no parficular order was given as dominanf we marked if as 'mixed'. 

The main word order of German and Arabic deserves a special menfion. In German, 
fhe posifioning of subjecf, objecf and verb is S5mfacfically defermined: main clauses 
wifhouf auxiliary verb are SVO, while subordinafe clauses and clauses confaining an 
auxiliary are SOV. A furfher complicafion, nof marked in Table is fhaf fhe German fi- 
nife verb musf be placed in second posifion, in which case fhe paffem becomes SAuxOV, 
wifh fhe objecf infervening between auxiliary and main verb. As regards Arabic, while 
the WATS classifies Modem Sfandard Arabic as VSO, fhe corpora fypically used in 
SMT show a very mixed disfribufion of VSO and SVO clauses P^|Carpuaf, Marfon, 
and Habash (201^ examined fhe Arabic-English Treebank and found fhaf, when fhe 
subjecf is expressed, if follows fhe verb in 70% of fhe cases, buf precedes if in 30%. 
Similarly, in fhe Pennsylvania Arabic Treebank, fhey found an order disfribufion of 67% 
VS and 33% SV. Besides frequency, if can be nofed fhaf fhe SVO senfences affesfed in 
fhese corpora are in general pragmafically neufral. We conjecfure fhaf fhis variabilify 
in Modern Sfandard Arabic may be due fo fhe effecf of spoken language variefies such 
as Egypfian, Gulf, Kuwaifi, Iraqi (all lisfed as SVO by fhe WALS), and Syrian (lisfed as 
VSO/SVO). For fhese reasons, we classify Arabic as a mixed VSO/SVO language. 

If is worfh nofing fhaf our seven-language sample covers fhe main word order fypes 
of fhe large majorify of fhe world languages: namely SOV, SVO and VSO (see Table|^. 

4.3 Word order differences 

Linguisfically mofivafed word order profiles can be very helpful fo anficipafe fhe kind 
of word reordering problems fhaf an SMT sysfem will have fo face. Clearly, fhese will 
also vary in relafion fo fhe fexf genre (wriffen news, speeches, efc.) and fo fhe frans- 
lafion's sfyle and degree of liferalify. However, we can reasonably expecf fhe S 5 mfacfic 
properfies of fwo languages fo defermine fhe general reordering characferisfics of fhaf 
pair. 

We will now analyze fhe reordering characferisfics of seven language pairs: English 
paired wifh fhe ofher six languages presenfed in Table as well as fhe French and Ara¬ 
bic pair. To fhis end, we propose fhe following analysis procedure. As a firsf indicafion 
of reordering complexify, we look af fhe main word order feafure (subjecf, objecf, verb). 
A difference af fhis level fypically resulfs in poor SMT performances. Then, we counf 
fhe fofal number of discordanf feafures. To simplify, if a parficular elemenf does nof 
exisf in a language (e. g. polar quesfion particles in English) we counf zero difference 
for fhaf feafure, and when one of fhe languages has a mixed order we counf a half 
difference. We insisf, however, on fhe qualifafive nafure of our analysis: numbers are 
only meaningful in combinafion wifh fhe lisf of specific discordanf feafures, as fhese 
have differenf impacf on word reordering. In parficular, we find if essenfial for SMT 
fo disfinguish befween clause-level and phrase-level differences (CDiff and PDiff) 
because fhe former accounf for mosf longer-range word movemenfs, and fhe latter 


16 VOS order is also admitted in Arabic, but only in specific contexts (e. g. when the object is expressed by a 
pronoun). 
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for the shorter. Thus, a language pair with only phrase-level discordant features is 
likely to be suitable for a PSMT approach, where reordering is managed through local 
distortion or inside translation units. On the contrary, the presence of many clause- 
level differences typically calls for a tree-based solution, either at preprocessing or at 
decoding time. As we will see, some pairs lay on the borderline, with only one or few 
clause-level differences. Finally, it should be noted that, even among features of the 
same group, some have more impact on SMT than others due to their frequency or to 
the average length of their constituents. For instance, the order of noun and genitive is 
more important than that of adjective and degree word. 


English and German [ Main order: different; CDiff=1.5; PDiff=0.5 ] 

The main word order of German is SVO or SOV according to the S 5 mtactic context 
(cf. Section [4.2| . German also differs from English with respect to the position 
of oblique phrases and that of the negation: both fixed in English but mixed in 
German. At the phrase level, German predominantly places the genitive after the 
noun, while English displays both orders. 

Thus, despite belonging to the same family branch (Indo-European/Germanic), 
this pair displays complex reordering patterns. Indeed, German-English reorder¬ 
ing has been widely studied in SMT and is still an open topic. At the Workshop 
on Statistical Machine Translation 2014 ([Bojar et al. 2014 1, a syntax-based string- 
to-tree SMT approach ( Williams et al. 2014) was wirming in both language direc¬ 
tions (official results excluding online systems). At the International Workshop on 
Spoken Language Translation 2014 ( Cettolo et al. 2014) , the best submission was 
a combination of PSMT with POS- and S 5 mtax-based preordering ([Slawik et alT 


2014| , string-to-tree S 5 mtax-based SMT and factored PSMT ( [Birch et al. 2014[ |. 


English and French [ Main order: same; CDiff: 0.5; PDiff: 1.5 

Most clause-level features have the same values in French as in English, except for 
the negation which is t 5 rpically expressed by two words in French: one preceding 
and one following the verbj^ At the phrase level, differences are found in the 
location of genitives and adjectives. Thus, English and French have very similar 
clause-level orders, but reordering is abundant at the local level. 

This is a case where reordering is mostly well handled by string-based PSMT. 
As a reference, the three top English-to-French WMT14 systems (official results 
excluding online systems), were all phrase-based. A similar trend was observed 
in the French-to-English track. 

English and Arabic [ Main order: different; CDiff: 0.5; PDiff: 2.5 ] 

The dominant Arabic order is VSO, followed by SVO (cf. Section |4^ . Apart from 
this important difference, all other clause-level features agree between Arabic and 
English. At the phrase level, differences are found in genitives, adjectives and 
degree words. 

As a result, reordering is overwhelmingly local but few crucial long-range re¬ 
orderings also regularly occur. Thus, this pair is challenging for PSMT but, at the 


same time, not well suited for a tree-based approach. As shown by Zollmann et 
al. (2008) and jBirch, Blunsom, and Osborne (2009|l, PSMT performs similarly or 


better than HSMT for the Arabic-to-English language pair. However, HSMT was 
shown to better cope with the reordering of VSO sentences ( [Bisazza 2013| . Pre¬ 
ordering of Arabic VSO sentences for translation into English has proved to be a 


17 Pre-verbal negation can be omitted in colloquial French. 
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particularly difficult task (Green, Sathi, and Manning 2009 Carpuat, Marton, and 
Habash 2010| and has inspired work on hybrid pre-ordering where multiple verb 


pre-orderings are fed to a PSMT decoder (Bisazza and Federico 2010 Andreas, 
[Habash, and Rambow 20lT||, see also Section 2.4 


English and Turkish [ Main order: different; CDiff: 5.5; PDiff: 1.5 ] 

Turkish is a good example of head-final language, except for the fact that it can 
employ both clause-final and clause-initial subordinatorsj^As a result, almost all 
clause-level features are discordant in this pair. At the phrase level, Turkish mainly 
differs from English for the use of postpositions instead of prepositions. 

Among our language pairs, this is one of the most difficult to reorder for an SMT 
system. The complex nature of its reordering phenomena suggests a good fit for 
tree-based SMT approaches, and indeed HSMT was shown to significantly out¬ 
perform PSMT between Turkish and English in both language directions ([Ruiz et 


al. 2012 [Yilmaz et al. 2013). However, state-of-the-art SMT quality in this language 


pair is still very low, mostly due to the agglutinative nature of Turkish which 
makes it difficult to tear apart word reordering issues from rich morphology 
issues. Attempting to address both issues in an English-to-Turkish factored PSMT 
system, Yeniterzi and Oflazer (2010| pre-process the parsed English side with 
a number of S5mtax-to-morphology mapping rules and costituent pre-ordering 
rules dealing with local and global reordering phenomena respectively. Only the 
former, though, resulted in better translation quality. 

English and Japanese [ Main order: different; CDiff: 6; PDiff: 1.5 ] 

Japanese is the protot 5 rpical example of head-final language. In this pair all clause- 
level features are discordant, while at the phrase level, Japanese differs from En¬ 
glish for the use of postpositions and the strictly head-final genitive construction. 
This pair, like the previous one, is extremely challenging for PSMT due to the hier¬ 
archical nature of its reordering phenomena and the high frequency of long-range 
word movements. Indeed, translation between English and Japanese has spurred 
a remarkable amount of work on pre-ordering, post-ordering and decoding-time 
reordering. In 2013 the PatentMT evaluation campaign of the NTCIR conference 
( [Goto et al. 2013a) saw rule-based and hybrid systems largely outperform the 
purely statistical ones in Japanese-to-English. The highest-ranked SMT submis¬ 
sion was actually a combination of three SMT systems including: a baseline PSMT 
method, a rule-based pre-ordering method, and a post-ordering method based 
on strrng-to-tree syntax-based SMT ( Sudoh et al. 2013) . Interestingly, the trends 
were different in the opposite translation direction, English-to-Japanese, where 
all rule-based MT systems were significantly outperformed by a PSMT system 
that performed pre-ordering of the English input with few manual rules for head 
finalization based on dependency parse trees ( jSudoh et al. 2013) . 

English and Chinese [ Main order: same; CDiff: 3.5; PDiff: 1 ] 

Despite belonging to the same main order t 5 q)e, these two languages differ in 
the positioning of oblique phrases, relative clauses, interrogative phrases and 
subordinating wordsj^ Moreover, word order variations are quite common in 


18 In Turkish, non-finite subordinate clauses are typically placed before the main clause and linked to it by a 
clause-final subordinator (e. g. ragmen/although), whereas finite subordinate clauses can be placed after 
the main clause and introduced by a clause-initial subordinator (e. g. ama/but). The former is dominant in 
written language. 

19 Subordi nating w ords in Chinese can occur at the beginning of the subordinate clause, at its end, or even 
inside it {Li 2008}. 
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Chinese to mark the topic of a sentence, i.e. what is being talked about. Comparing 
the two languages at the phrase level, we find partial disagreemenf in fhe use of 
genitive and adposifions (Chinese has bofh preposifions and posfposifions). 

Thus, fhis pair foo is characferized by very complex reordering, hardly man¬ 
ageable by a PSMT sysfem. This is confirmed by a number of empirical resulfs 
showing fhaf free-based approaches (parficularly HSMT) consisfently outperform 
PSMT in Chinese-fo-English evaluafions ([Zollmann ef al. 2008} [Birch, Blunsom, 


and Osborne 200^. It is worth noting that translation between Chinese and En¬ 


glish has been the main motivation and test bed for fhe developmenf of HSMT. 

French and Arabic [ Main order: differenf; CDiff: 1.5; PDiff: 1 ] 

Af fhe clause level, fhis pair differs in main word order (SVO versus VSO or SVO) 
like fhe English-Arabic pair, buf also in fhe order of negation and verb. On fhe 
ofher hand, phrase-level order is nofably more similar, wifh only one discordanf 
feafure of minor imporfance (adjecfive and degree word). 

Less research was published on fhis language pair. Neverfheless, Hasan and Ney 
(2008 1 and Schwenk and Senellarf (2009| chose a PSMT approach fo experimenf 


wifh an Arabic-fo-French fask. 


Figure l^illusfrafes fhe reordering characferisfics of fhree language pairs by means 
of sentence examples that were automatically word-aligned with GIZA-n- (jOch and 
Ney 200^ (intersection of direcf and inverse alignmenfs). On fhe firsf row, we see two 
English-German senfence pairs: in bofh cases, mosf of fhe poinfs lie close fo fhe diagonal 
representing an overall monofonic franslafion, whereas few isolafed poinfs denofe fhe 
very long-range reordering of verbs. Similarly, in fhe fwo English-Arabic senfence pairs, 
we mosfly observe local reorderings, wifh fhe exception of few isolafed poinfs corre¬ 
sponding fo fhe Arabic clause-initial verbs. Finally, fhe fwo Turkish-English examples 
display global reordering, due fo fhe high number of clause-level order differences. 


Where possible, if is inferesfing fo relafe our analysis wifh previously published 
measures of reordering based on parallel dafa. To our knowledge, fhe mosf compre¬ 
hensive resulfs of fhis kind are reported by Birch (2011| , who formulates reordering 
as a binary process occurring between two blocks that are adjacent in the source (cf. 
ITG constraints in Section 2.11. Here, the general amount of reordering in a language 
pair is esfimafed by fhe RQuantify, defined as fhe sum of fhe spans of all fhe reordered 
blocks on fhe fargef side, normalized by fhe lengfh of fhe fargef senfence and aver¬ 
aged over a corpus. Based on fhe Europarl corpus ( Koehn 2002) and aufomafic word 
alignmenfs, Birch (2011) reporfs average RQuantify values of 0.586/0.608 in English-fo- 
German/German-fo-English, versus only 0.402/0.395 in English-fo-French/French-fo- 
English. The manually-aligned GALE corpus (LDG2006E93) is insfead used fo measure 
the distribution of reordering widfhs, defined as fhe sum of fhe swapped blocks' far¬ 
gef spans. Widfhs are binned info short (2-4 words), medium (5-8), and long (>8). In 
Chinese-to-English there are about 0.8/0.9/0.9 short/medium/long reordered blocks 
per sentence, while in Arabic-to-English there are 1.1/0.4/0.2 short/medium/long 
reordered blocks per sentence. These figures align nicely wifh our classificafion of 
phrase- and clause-level differences, which we have relafed fo longer and shorfer- 
range reordering respectively: Ghinese-fo-English (PDiff: 1, CDiff: 3.5) displays much 
more reordering overall, while Arabic-fo-English (PDiff: 2.5, CDiff: 0.5) has more shorf 
reorderings buf much less medium and shorf. 

The advanfage of using our proposed analysis is fhaf if can be easily exfended 
fo ofher language pairs fhanks fo fhe wide coverage of WALS, whereas dafa-driven 
analyses depend on fhe availabilify of high-qualify word-aligned parallel corpora. 
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English and German: 




English and Arabic: 


s I 


1 H . I 'h U S I ! g 5 




■; . v: 'i' a';;,; tv,',.'.. .r 

!J.ihd{.=i.,.diidf.JdLLi!Lhii 


.dhdi.JdhLiiLidiJi 

English and Turkish: 


ii.iLdL,,. 


h.inh-liiisiii 






Eigure 8: Word-alignment matrices of sentence pairs taken from three parallel news 
corpora: the NlST-MT-08 Arabic-English evaluation benchmark, the WMT-10 German- 
English training corpus, and the Turkish-English South European Times corpus (Tyers 
and Alperen 2010). English is always on the x axis. 
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5. Discussion and conclusions 

We have provided a comprehensive overview of how the word reordering problem is 
modeled within different string-based and tree-based SMT frameworks, and as a stand¬ 
alone task. To summarize, string-based SMT considers all permutations of the source 
sentence and relies on separate reordering models to score them. On the other hand, 
tree-based SMT tightly couples reordering to translation and, during decoding, only or 
mostly considers word permutations that are licensed by the learnt translation model. 
In practice, both approaches apply general heuristic constraints on the maximum re¬ 
ordering width to avoid explosion of the search space. 

The main weakness of a string-based approach like phrase-based SMT (PSMT) with 
regard to reordering lies in its coarse definition of the reordering search space. In this 
framework, relaxing the distortion limit means dramatically increasing the size of the 
search space, making the reordering model's task extremely complex and intensifying 
the risk of both search and model errors. As a result, PSMT is generally good at handling 
local reordering but largely fails to capture long-range reordering phenomena. 

As for tree-based SMT, a distinction must be made between methods that extract 
hierarchical structure directly from parallel data and methods that rely on S 5 mtactic 
annotation provided by pre-trained monolingual parsers. A prominent example of the 
former is hierarchical phrase-based SMT (HSMT), which models reordering via partially 
lexicalized translation rules. While this results in a more principled definition of the 
reordering search space, HSMT lacks the ability to generalize the learnt reordering 
patterns from specific lexical clues to whole word or phrase categories. 

Finally, reordering may be constrained by S 5 mtactic information in the source or 
target language, or both. When syntax is used in the source language, reordering is 
performed by transforming a given parse tree of the input sentence. When S 5 mtax is 
used in the target language, reordering is allowed only if resulting in a grammatically 
valid target tree fragment. S 5 mtactic information is adopted by both S 5 mtax-based SMT, 
where the tree is reordered and translated simultaneously, and by S 5 mtactic pre-ordering 
(or post-ordering) methods, where the tree is transformed before (or after) translation. 
The success of these approaches largely depends on the degree of isomorphism of 
the modeled language pair, as well as on the parser's performance, which can vary 
substantially across languages. 

After describing how word reordering is modeled in SMT, we have questioned why 
different language pairs appear to need different reordering modeling solutions. To 
answer this question, we have outlined the word order profiles of seven widely spoken 
languages, based on a large body of linguistic knowledge. Then we have examined their 
pairwise differences in detail. Finally, we have used these differences to interpret the 
empirical findings of previous work that evaluated various SMT reordering techniques 
in those language pairs. 

We conclude from our analysis that a few linguistic facts can be very useful to pre¬ 
dict the reordering characteristics of a language pair and to select the SMT approach that 
best suits them. In particular, string-based PSMT is preferable for language pairs with 
only constituent-level differences, like French-English, as these mostly imply short or 
medium-range reordering patterns that can be captured by local distortion. On the other 
hand, language pairs with many clause-level order differences (e. g. Japanese-English, 
Turkish-English, Chinese-English) are best handled by tree-based SMT or S 5 mtax-based 
pre-/post-ordering approaches that can handle complex, hierarchical reordering pat¬ 
terns. While this may seem obvious, we notice that, in the literature, the choice of an 
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optimal SMT framework for a new franslafion fask is offen driven by cosfly empirical 
frials rafher fhan by linguisfic knowledge. Finally, fhe pairs wifh mosfly consfifuenf- 
level differences and only one or few clause-level differences (e. g. German-English 
and Arabic-English) do nof fif well info eifher cafegory. In senfences wifhouf global 
reordering, HSMT can underperform PSMT, likely due fo fhe much larger search space 
explored. Af fhe same fime, applying PSMT fo such pairs wifh heurisfic reordering 
consfrainfs can lead fo sysfemafic errors in fhe posifioning of imporfanf elemenfs of 
fhe senfence, such as verbs. Nof surprisingly, fhese language pairs have been fhe objecf 
of a fair amounf work aiming af refining fhe reordering space of bofh PSMT and HSMT. 

Our word order analysis can be easily exfended fo ofher language pairs using fhe 
mefhodology presenfed in Secfion|^ 

In conclusion, finding a definifive solufion fo fhe problem of word reordering 
implies answering fhe fundamenfal research quesfions of SMT: Is sfrucfure needed fo 
franslafe? If so, whaf kind of sfrucfure and how should if be used? A growing parf of fhe 
research communify has converged on a posifive answer fo fhe former quesfion, buf fhe 
laffer remains open fo dafe. While fhe field keeps evolving around fhese quesfions, SMT 
has already reached fhe sfage of applied language fechnology. We hope fhis survey will 
provide pracfical guidelines fo fhe sysfem developers of foday and, af fhe same fime, 
good scienfific references fo fhe researchers elaborafing fhe solufions of fomorrow. 
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